Clinical Decision Velocity is Increased when Meta-search Filters Enhance an Evidence Retrieval System

Objective

To test whether the use of an evidence retrieval system that uses clinically targeted meta-search filters can enhance the rate at which clinicians make correct decisions, reduce the effort involved in locating evidence, and provide an intuitive match between clinical tasks and search filters.

Design

A laboratory experiment under controlled conditions asked 75 clinicians to answer eight randomly sequenced clinical questions, using one of two randomly assigned search engines. The first search engine Quick Clinical (QC) was equipped with meta-search filters (the combined use of meta-search and search filters) designed to answer typical clinical questions e.g., treatment, diagnosis, and the second ‘library model’ system (LM) offered free access to an identical evidence set with no filter support.

Measurements

Changes in clinical decision making were measured by the proportion of correct post-search answers provided to questions, the time taken to answer questions, and the number of searches and links to documents followed in a search session. The intuitive match between meta-search filters and clinical tasks was measured by the proportion and distribution of filters selected for individual clinical questions.

Results

Clinicians in the two groups performed equally well pre-search. Post search answers improved overall by 21%, with 52.2% of answers correct with QC and 54.7% with LM (χ² = 0.33, df = 1, p > 0.05). Users of QC obtained a significantly greater percentage of their correct answers within the first two minutes of searching compared to LM users (QC 58.2%; LM 32.9%; χ² = 19.203, df = 1, p < 0.001). There was a statistical difference for QC and LM survival curves, which plotted overall time to answer questions, irrespective of answer (Wilcoxon, p = 0.019) and for the average time to provide a correct answer (Wilcoxon, p = 0.006). The QC system users conducted significantly fewer searches per scenario (m = 3.0 SD = 1.15 versus m = 5.5 SD1.97, t = 6.63, df = 72, p = 0.0001). Clinicians using the QC system followed fewer document links than did those who used LM (respectively 3.9 links SD = 1.20 versus 4.7 links SD = 1.79, t = 2.13, df = 72, p = 0.0368). In 6 of the 8 questions, two meta-search filters accounted for 89% or more of clinicians'' first choice, suggesting the choice of filter intuitively matched the clinical decision task at hand.

Conclusions

Meta-search filters result in clinicians arriving at answers more quickly than unconstrained searches across information sources, and appear to increase the rate with which correct decisions are made. In time restricted clinical settings meta-search filters may thus improve overall decision accuracy, as fewer searches that could otherwise lead to a correct answer are abandoned. Meta-search filters appear to be intuitive to use, suggesting that the simplicity of the user model would fit very well into clinical settings.     

Towards semantic search and inference in electronic medical records: An approach using concept--based information retrieval

Background

This paper presents a novel approach to searching electronic medical records that is based on concept matching rather than keyword matching.

Aim

The concept-based approach is intended to overcome specific challenges we identified in searching medical records.

Method

Queries and documents were transformed from their term-based originals into medical concepts as defined by the SNOMED-CT ontology.

Results

Evaluation on a real-world collection of medical records showed our concept-based approach outperformed a keyword baseline by 25% in Mean Average Precision.

Conclusion

The concept-based approach provides a framework for further development of inference based search systems for dealing with medical data.     

A Day in the Life of PubMed: Analysis of a Typical Day’s Query Log

Objective

To characterize PubMed usage over a typical day and compare it to previous studies of user behavior on Web search engines.

Design

We performed a lexical and semantic analysis of 2,689,166 queries issued on PubMed over 24 consecutive hours on a typical day.

Measurements

We measured the number of queries, number of distinct users, queries per user, terms per query, common terms, Boolean operator use, common phrases, result set size, MeSH categories, used semantic measurements to group queries into sessions, and studied the addition and removal of terms from consecutive queries to gauge search strategies.

Results

The size of the result sets from a sample of queries showed a bimodal distribution, with peaks at approximately 3 and 100 results, suggesting that a large group of queries was tightly focused and another was broad. Like Web search engine sessions, most PubMed sessions consisted of a single query. However, PubMed queries contained more terms.

Conclusion

PubMed’s usage profile should be considered when educating users, building user interfaces, and developing future biomedical information retrieval systems.     

Effects of librarian-provided services in healthcare settings: a systematic review

Objective

To assess the effects of librarian-provided services in healthcare settings on patient, healthcare provider, and researcher outcomes.

Materials and methods

Medline, CINAHL, ERIC, LISA (Library and Information Science Abstracts), and the Cochrane Central Register of Controlled Trials were searched from inception to June 2013. Studies involving librarian-provided services for patients encountering the healthcare system, healthcare providers, or researchers were eligible for inclusion. All librarian-provided services in healthcare settings were considered as an intervention, including hospitals, primary care settings, or public health clinics.

Results

Twenty-five articles fulfilled our eligibility criteria, including 22 primary publications and three companion reports. The majority of studies (15/22 primary publications) examined librarians providing instruction in literature searching to healthcare trainees, and measured literature searching proficiency. Other studies analyzed librarian-provided literature searching services and instruction in question formulation as well as the impact of librarian-provided services on patient length of stay in hospital. No studies were found that investigated librarians providing direct services to researchers or patients in healthcare settings.

Conclusions

Librarian-provided services directed to participants in training programs (eg, students, residents) improve skills in searching the literature to facilitate the integration of research evidence into clinical decision-making. Services provided to clinicians were shown to be effective in saving time for health professionals and providing relevant information for decision-making. Two studies indicated patient length of stay was reduced when clinicians requested literature searches related to a patient''s case.     

Identifying Diagnostic Studies in MEDLINE: Reducing the Number Needed to Read

Objectives. The search filters in PubMed have become a cornerstone in information retrieval in evidence-based practice. However, the filter for diagnostic studies is not fully satisfactory, because sensitive searches have low precision. The objective of this study was to construct and validate better search strategies to identify diagnostic articles recorded on MEDLINE with special emphasis on precision.Design. A comparative, retrospective analysis was conducted. Four medical journals were hand-searched for diagnostic studies published in 1989 and 1994. Four other journals were hand-searched for 1999. The three sets of studies identified were used as gold standards. A new search strategy was constructed and tested using the 1989-subset of studies and validated in both the 1994 and 1999 subsets. We identified candidate text words for search strategies using a word frequency analysis of the abstracts. According to the frequency of identified terms, searches were run for each term independently. The sensitivity, precision, and number needed to read (1/precision) of every candidate term were calculated. Terms with the highest sensitivity × precision product were used as free text terms in combination with the MeSH term “SENSITIVITY AND SPECIFICITY” using the Boolean operator OR. In the 1994 and 1999 subsets, we performed head-to-head comparisons of the currently available PubMed filter with the one we developed.Measurements. The sensitivity, precision and the number needed to read (1/precision) were measured for different search filters.Results. The most frequently occurring three truncated terms (diagnos*; predict* and accura*) in combination with the MeSH term “SENSITIVITY AND SPECIFICITY” produced a sensitivity of 98.1 percent (95% confidence interval: 89.9–99.9%) and a number needed to read of 8.3 (95% confidence interval: 6.7–11.3%). In direct comparisons of the new filter with the currently available one in PubMed using the 1994 and 1999 subsets, the new filter achieved better precision (12.0% versus 8.2% in 1994 and 5.0% versus 4.3% in 1999. The 95% confidence intervals for the differences range from 0.05% to 7.5% (p = 0.041) and –1.0% to 2.3% (p = 0.45), respectively). The new filter achieved slightly better sensitivities than the currently available one in both subsets, namely 98.1 and 96.1% (p = 0.32) versus 95.1 and 88.8% (p = 0.125).Conclusions. The quoted performance of the currently available filter for diagnostic studies in PubMed may be overstated. It appears that even single external validation may lead to over optimistic views of a filter’s performance. Precision appears to be more unstable than sensitivity. In terms of sensitivity, our filter for diagnostic studies performed slightly better than the currently available one and it performed better with regards to precision in the 1994 subset. Additional research is required to determine whether these improvements are beneficial to searches in practice.Biomedical databases are important sources of evidence in medical practice. However, information retrieval in such databases can become very time-consuming because searches that are likely to identify all relevant information also find many irrelevant articles.In recent years researchers have adopted various approaches in the development of search strategies to selectively retrieve different types of studies (therapy, prognosis, diagnosis and etiology) and different study designs.^1,2 Search strategies targeted at diagnostic studies have also been developed.^1,3–4 The most commonly used filter for diagnostic studies is almost certainly the one now publicly available in PubMed (Clinical Queries),⁵ which based on the work of Haynes and coworkers.¹ Their search filter with emphasis on sensitivity achieved a cross-validated mean sensitivity of approximately 87% combined with a (non–cross-validated) mean precision of approximately 8%.Compared with the filter for therapeutic studies, the diagnostic filter’s precision in particular is much lower. The main reason for this difference may be the inconsistent terminology used in diagnostic studies making them difficult to index and retrieve in electronic databases.In view of this high false-positive rate, we wondered if it would be possible to develop a more precise search strategy for selecting publications on diagnostic test evaluations without losing sensitivity. Our objective was to develop, test and validate a generic search strategy for the detection of diagnostic articles recorded on MEDLINE that can be applied in any diagnostic field in Medicine.     

A Comparative Evaluation of Full-text,Concept-based,and Context-sensitive Search

Objectives

Study comparatively (1) concept-based search, using documents pre-indexed by a conceptual hierarchy; (2) context-sensitive search, using structured, labeled documents; and (3) traditional full-text search. Hypotheses were: (1) more contexts lead to better retrieval accuracy; and (2) adding concept-based search to the other searches would improve upon their baseline performances.

Design

Use our Vaidurya architecture, for search and retrieval evaluation, of structured documents classified by a conceptual hierarchy, on a clinical guidelines test collection.

Measurements

Precision computed at different levels of recall to assess the contribution of the retrieval methods. Comparisons of precisions done with recall set at 0.5, using t-tests.

Results

Performance increased monotonically with the number of query context elements. Adding context-sensitive elements, mean improvement was 11.1% at recall 0.5. With three contexts, mean query precision was 42% ± 17% (95% confidence interval [CI], 31% to 53%); with two contexts, 32% ± 13% (95% CI, 27% to 38%); and one context, 20% ± 9% (95% CI, 15% to 24%). Adding context-based queries to full-text queries monotonically improved precision beyond the 0.4 level of recall. Mean improvement was 4.5% at recall 0.5. Adding concept-based search to full-text search improved precision to 19.4% at recall 0.5.

Conclusions

The study demonstrated usefulness of concept-based and context-sensitive queries for enhancing the precision of retrieval from a digital library of semi-structured clinical guideline documents. Concept-based searches outperformed free-text queries, especially when baseline precision was low. In general, the more ontological elements used in the query, the greater the resulting precision.     

From web search to healthcare utilization: privacy-sensitive studies from mobile data

Objective

We explore relationships between health information seeking activities and engagement with healthcare professionals via a privacy-sensitive analysis of geo-tagged data from mobile devices.

Materials and methods

We analyze logs of mobile interaction data stripped of individually identifiable information and location data. The data analyzed consist of time-stamped search queries and distances to medical care centers. We examine search activity that precedes the observation of salient evidence of healthcare utilization (EHU) (ie, data suggesting that the searcher is using healthcare resources), in our case taken as queries occurring at or near medical facilities.

Results

We show that the time between symptom searches and observation of salient evidence of seeking healthcare utilization depends on the acuity of symptoms. We construct statistical models that make predictions of forthcoming EHU based on observations about the current search session, prior medical search activities, and prior EHU. The predictive accuracy of the models varies (65%–90%) depending on the features used and the timeframe of the analysis, which we explore via a sensitivity analysis.

Discussion

We provide a privacy-sensitive analysis that can be used to generate insights about the pursuit of health information and healthcare. The findings demonstrate how large-scale studies of mobile devices can provide insights on how concerns about symptomatology lead to the pursuit of professional care.

Conclusion

We present new methods for the analysis of mobile logs and describe a study that provides evidence about how people transition from mobile searches on symptoms and diseases to the pursuit of healthcare in the world.     

Automated medical literature retrieval

Background

The constantly growing publication rate of medical research articles puts increasing pressure on medical specialists who need to be aware of the recent developments in their field. The currently used literature retrieval systems allow researchers to find specific papers; however the search task is still repetitive and time-consuming.

Aim

In this paper we describe a system that retrieves medical publications by automatically generating queries based on data from an electronic patient record. This allows the doctor to focus on medical issues and provide an improved service to the patient, with higher confidence that it is underpinned by current research.

Method

Our research prototype automatically generates query terms based on the patient record and adds weight factors for each term. Currently the patient’s age is taken into account with a fuzzy logic derived weight, and terms describing blood-related anomalies are derived from recent blood test results. Conditionally selected homonyms are used for query expansion.The query retrieves matching records from a local index of PubMed publications and displays results in descending relevance for the given patient. Recent publications are clearly highlighted for instant recognition by the researcher.

Results

Nine medical specialists from the Royal Adelaide Hospital evaluated the system and submitted pre-trial and post-trial questionnaires. Throughout the study we received positive feedback as doctors felt the support provided by the prototype was useful, and which they would like to use in their daily routine.

Conclusion

By supporting the time-consuming task of query formulation and iterative modification as well as by presenting the search results in order of relevance for the specific patient, literature retrieval becomes part of the daily workflow of busy professionals.     

The effectiveness of interventions using electronic reminders to improve adherence to chronic medication: a systematic review of the literature

Background

Many patients experience difficulties in adhering to long-term treatment. Although patients'' reasons for not being adherent are diverse, one of the most commonly reported barriers is forgetfulness. Reminding patients to take their medication may provide a solution. Electronic reminders (automatically sent reminders without personal contact between the healthcare provider and patient) are now increasingly being used in the effort to improve adherence.

Objective

To examine the effectiveness of interventions using electronic reminders in improving patients'' adherence to chronic medication.

Methods

A comprehensive literature search was conducted in PubMed, Embase, PsycINFO, CINAHL and Cochrane Central Register of Controlled Trials. Electronic searches were supplemented by manual searching of reference lists and reviews. Two reviewers independently screened all citations. Full text was obtained from selected citations and screened for final inclusion. The methodological quality of studies was assessed.

Results

Thirteen studies met the inclusion criteria. Four studies evaluated short message service (SMS) reminders, seven audiovisual reminders from electronic reminder devices (ERD), and two pager messages. Best evidence synthesis revealed evidence for the effectiveness of electronic reminders, provided by eight (four high, four low quality) studies showing significant effects on patients'' adherence, seven of which measured short-term effects (follow-up period <6 months). Improved adherence was found in all but one study using SMS reminders, four studies using ERD and one pager intervention. In addition, one high quality study using an ERD found subgroup effects.

Conclusion

This review provides evidence for the short-term effectiveness of electronic reminders, especially SMS reminders. However, long-term effects remain unclear.     

The impact of the electronic medical record on structure,process, and outcomes within primary care: a systematic review of the evidence

Background

The electronic medical record (EMR)/electronic health record (EHR) is becoming an integral component of many primary-care outpatient practices. Before implementing an EMR/EHR system, primary-care practices should have an understanding of the potential benefits and limitations.

Objective

The objective of this study was to systematically review the recent literature around the impact of the EMR/EHR within primary-care outpatient practices.

Materials and methods

Searches of Medline, EMBASE, CINAHL, ABI Inform, and Cochrane Library were conducted to identify articles published between January 1998 and January 2010. The gray literature and reference lists of included articles were also searched. 30 studies met inclusion criteria.

Results and discussion

The EMR/EHR appears to have structural and process benefits, but the impact on clinical outcomes is less clear. Using Donabedian''s framework, five articles focused on the impact on healthcare structure, 21 explored healthcare process issues, and four focused on health-related outcomes.     

Patient access to medical records and healthcare outcomes: a systematic review

Objectives

We conducted a systematic review to determine the effect of providing patients access to their medical records (electronic or paper-based) on healthcare quality, as defined by measures of safety, effectiveness, patient-centeredness, timeliness, efficiency, and equity.

Methods

Articles indexed in PubMed from January 1970 to January 2012 were reviewed. Twenty-seven English-language controlled studies were included. Outcomes were categorized as measures of effectiveness (n=19), patient-centeredness (n=16), and efficiency (n=2); no study addressed safety, timeliness, or equity.

Results

Outcomes were equivocal with respect to several aspects of effectiveness and patient-centeredness. Efficiency outcomes in terms of frequency of in-person and telephone encounters were mixed. Access to health records appeared to enhance patients’ perceptions of control and reduced or had no effect on patient anxiety.

Conclusion

Although few positive findings generally favored patient access, the literature is unclear on whether providing patients access to their medical records improves quality.     

Automated concept-level information extraction to reduce the need for custom software and rules development

Objective

Despite at least 40 years of promising empirical performance, very few clinical natural language processing (NLP) or information extraction systems currently contribute to medical science or care. The authors address this gap by reducing the need for custom software and rules development with a graphical user interface-driven, highly generalizable approach to concept-level retrieval.

Materials and methods

A ‘learn by example’ approach combines features derived from open-source NLP pipelines with open-source machine learning classifiers to automatically and iteratively evaluate top-performing configurations. The Fourth i2b2/VA Shared Task Challenge''s concept extraction task provided the data sets and metrics used to evaluate performance.

Results

Top F-measure scores for each of the tasks were medical problems (0.83), treatments (0.82), and tests (0.83). Recall lagged precision in all experiments. Precision was near or above 0.90 in all tasks.

Discussion

With no customization for the tasks and less than 5 min of end-user time to configure and launch each experiment, the average F-measure was 0.83, one point behind the mean F-measure of the 22 entrants in the competition. Strong precision scores indicate the potential of applying the approach for more specific clinical information extraction tasks. There was not one best configuration, supporting an iterative approach to model creation.

Conclusion

Acceptable levels of performance can be achieved using fully automated and generalizable approaches to concept-level information extraction. The described implementation and related documentation is available for download.     

Automatically extracting sentences from Medline citations to support clinicians’ information needs

Objective

Online health knowledge resources contain answers to most of the information needs raised by clinicians in the course of care. However, significant barriers limit the use of these resources for decision-making, especially clinicians’ lack of time. In this study we assessed the feasibility of automatically generating knowledge summaries for a particular clinical topic composed of relevant sentences extracted from Medline citations.

Methods

The proposed approach combines information retrieval and semantic information extraction techniques to identify relevant sentences from Medline abstracts. We assessed this approach in two case studies on the treatment alternatives for depression and Alzheimer''s disease.

Results

A total of 515 of 564 (91.3%) sentences retrieved in the two case studies were relevant to the topic of interest. About one-third of the relevant sentences described factual knowledge or a study conclusion that can be used for supporting information needs at the point of care.

Conclusions

The high rate of relevant sentences is desirable, given that clinicians’ lack of time is one of the main barriers to using knowledge resources at the point of care. Sentence rank was not significantly associated with relevancy, possibly due to most sentences being highly relevant. Sentences located closer to the end of the abstract and sentences with treatment and comparative predications were likely to be conclusive sentences. Our proposed technical approach to helping clinicians meet their information needs is promising. The approach can be extended for other knowledge resources and information need types.     

Knowledge-based Methods to Help Clinicians Find Answers in MEDLINE

Objectives

Large databases of published medical research can support clinical decision making by providing physicians with the best available evidence. The time required to obtain optimal results from these databases using traditional systems often makes accessing the databases impractical for clinicians. This article explores whether a hybrid approach of augmenting traditional information retrieval with knowledge-based methods facilitates finding practical clinical advice in the research literature.

Design

Three experimental systems were evaluated for their ability to find MEDLINE citations providing answers to clinical questions of different complexity. The systems (SemRep, Essie, and CQA-1.0), which rely on domain knowledge and semantic processing to varying extents, were evaluated separately and in combination. Fifteen therapy and prevention questions in three categories (general, intermediate, and specific questions) were searched. The first 10 citations retrieved by each system were randomized, anonymized, and evaluated on a three-point scale. The reasons for ratings were documented.

Measurements

Metrics evaluating the overall performance of a system (mean average precision, binary preference) and metrics evaluating the number of relevant documents in the first several presented to a physician were used.

Results

Scores (mean average precision = 0.57, binary preference = 0.71) for fusion of the retrieval results of the three systems are significantly (p < 0.01) better than those for any individual system. All three systems present three to four relevant citations in the first five for any question type.

Conclusion

The improvements in finding relevant MEDLINE citations due to knowledge-based processing show promise in assisting physicians to answer questions in clinical practice.     

Metrics associated with NIH funding: a high-level view

Objective

To introduce the availability of grant-to-article linkage data associated with National Institutes of Health (NIH) grants and to perform a high-level analysis of the publication outputs and impacts associated with those grants.

Design

Articles were linked to the grants they acknowledge using the grant acknowledgment strings in PubMed using a parsing and matching process as embodied in the NIH Scientific Publication Information Retrieval & Evaluation System system. Additional data from PubMed and citation counts from Scopus were added to the linkage data. The data comprise 2 572 576 records from 1980 to 2009.

Results

The data show that synergies between NIH institutes are increasing over time; 29% of current articles acknowledge grants from multiple institutes. The median time lag to publication for a new grant is 3 years. Each grant contributes to approximately 1.7 articles per year, averaged over all grant types. Articles acknowledging US Public Health Service (PHS, which includes NIH) funding are cited twice as much as US-authored articles acknowledging no funding source. Articles acknowledging both PHS funding and a non-US government funding source receive on average 40% more citations that those acknowledging PHS funding sources alone.

Conclusion

The US PHS is effective at funding research with a higher-than-average impact. The data are amenable to further and much more detailed analysis.     

An analysis of Malawi's publication productivity

Background

Publications productivity, the number of scientific articles published, is a measure of a country''s scientific output. If measured carefully it can be a useful indicator that describes a countries'' research activity. Our objective was to analyze trends in publications originating from Malawi between 1996 – 2006.

Methods

The MEDLINE/PubMed database, a registry of articles from over 5,000 scientific journals was searched for articles originating from Malawi between 1996 – 2006 by typing Malawi in the author affiliation search field. A review of abstracts was performed to determine health field and origin of first author — Malawian vs foreign.

Results

506 articles were retrieved of which 489 were on health. 15.5% on TB, 14.5% on HIV and AIDS, 11.2% on infectious disease, 7.2% on TB and HIV, 7.2% on Malaria. 20.9% of the authors were of Malawian origin and Tropical Doctor was the journal that had the most articles originating from Malawi. The number of articles published from Malawi has grown by 106% in the past ten years.

Conclusions

Our results suggest there is growth in scientific publishing in Malawi but the main contribution is from foreign researchers residing in Malawi. More needs to be done to promote publishing by Malawian authors.