Research and applications: Resident physicians as human information systems: sources yet seekers

Objective

To characterize question types that residents received on overnight shifts and what information sources were used to answer them.

Materials and Methods

Across 30 overnight shifts, questions asked of on-call senior residents, question askers’ roles, and residents’ responses were documented. External sources were noted.

Results

158 of 397 questions (39.8%) related to the plan of care, 53 (13.4%) to medical knowledge, 48 (12.1%) to taskwork knowledge, and 44 (11.1%) to the current condition of patients. For 351 (88.4%) questions residents provided specific, direct answers or visited the patient. For 16 of these, residents modeled or completed the task. For 216 questions, residents used previous knowledge or their own clinical judgment. Residents solicited external information sources for 118 questions and only a single source for 77 (65.3%) of them. For the 118, most questions concerned either the plan of care or the patient''s current condition and were asked by interns and nurses (those with direct patient care responsibilities).

Discussion

Resident physicians serve as an information system and they often specifically answer the question using previous knowledge or their own clinical judgment, suggesting that askers are contacting an appropriately knowledgeable person. However, they do need to access patient information such as the plan of care. They also serve an educator role and answer many knowledge-related questions.

Conclusions

As synchronous verbal communications continue to be important pathways for information flow, informaticians need to consider the relationship between such communications and workflow in the development of healthcare support tools.     

Retrieval of diagnostic and treatment studies for clinical use through PubMed and PubMed's Clinical Queries filters

Objective

Clinical Queries filters were developed to improve the retrieval of high-quality studies in searches on clinical matters. The study objective was to determine the yield of relevant citations and physician satisfaction while searching for diagnostic and treatment studies using the Clinical Queries page of PubMed compared with searching PubMed without these filters.

Materials and methods

Forty practicing physicians, presented with standardized treatment and diagnosis questions and one question of their choosing, entered search terms which were processed in a random, blinded fashion through PubMed alone and PubMed Clinical Queries. Participants rated search retrievals for applicability to the question at hand and satisfaction.

Results

For treatment, the primary outcome of retrieval of relevant articles was not significantly different between the groups, but a higher proportion of articles from the Clinical Queries searches met methodologic criteria (p=0.049), and more articles were published in core internal medicine journals (p=0.056). For diagnosis, the filtered results returned more relevant articles (p=0.031) and fewer irrelevant articles (overall retrieval less, p=0.023); participants needed to screen fewer articles before arriving at the first relevant citation (p<0.05). Relevance was also influenced by content terms used by participants in searching. Participants varied greatly in their search performance.

Discussion

Clinical Queries filtered searches returned more high-quality studies, though the retrieval of relevant articles was only statistically different between the groups for diagnosis questions.

Conclusion

Retrieving clinically important research studies from Medline is a challenging task for physicians. Methodological search filters can improve search retrieval.     

Design and evaluation of a wireless electronic health records system for field care in mass casualty settings

Background

There is growing interest in the use of technology to enhance the tracking and quality of clinical information available for patients in disaster settings. This paper describes the design and evaluation of the Wireless Internet Information System for Medical Response in Disasters (WIISARD).

Materials and methods

WIISARD combined advanced networking technology with electronic triage tags that reported victims'' position and recorded medical information, with wireless pulse-oximeters that monitored patient vital signs, and a wireless electronic medical record (EMR) for disaster care. The EMR system included WiFi handheld devices with barcode scanners (used by front-line responders) and computer tablets with role-tailored software (used by managers of the triage, treatment, transport and medical communications teams). An additional software system provided situational awareness for the incident commander. The WIISARD system was evaluated in a large-scale simulation exercise designed for training first responders. A randomized trial was overlaid on this exercise with 100 simulated victims, 50 in a control pathway (paper-based), and 50 in completely electronic WIISARD pathway. All patients in the electronic pathway were cared for within the WIISARD system without paper-based workarounds.

Results

WIISARD reduced the rate of the missing and/or duplicated patient identifiers (0% vs 47%, p<0.001). The total time of the field was nearly identical (38:20 vs 38:23, IQR 26:53–1:05:32 vs 18:55–57:22).

Conclusion

Overall, the results of WIISARD show that wireless EMR systems for care of the victims of disasters would be complex to develop but potentially feasible to build and deploy, and likely to improve the quality of information available for the delivery of care during disasters.     

Information needs of case managers caring for persons living with HIV

Objective

The goals of this study were to explore the information needs of case managers who provide services to persons living with HIV (PLWH) and to assess the applicability of the Information Needs Event Taxonomy in a new population.

Design

The study design was observational with data collection via an online survey.

Measurements

Responses to open-ended survey questions about the information needs of case managers (n=94) related to PLWH of three levels of care complexity were categorized using the Information Needs Event Taxonomy.

Results

The most frequently identified needs were related to patient education resources (33%), patient data (23%), and referral resources (22%) accounting for 79% of all (N=282) information needs.

Limitations

Study limitations include selection bias, recall bias, and a relatively narrow focus of the study on case-manager information needs in the context of caring for PLWH.

Conclusion

The study findings contribute to the evidence base regarding information needs in the context of patient interactions by: (1) supporting the applicability of the Information Needs Event Taxonomy and extending it through addition of a new generic question; (2) providing a foundation for the addition of context-specific links to external information resources within information systems; (3) applying a new approach for elicitation of information needs; and (4) expanding the literature regarding addressing information needs in community-based settings for HIV services.     

The Evaluation of a Temporal Reasoning System in Processing Clinical Discharge Summaries

Context

TimeText is a temporal reasoning system designed to represent, extract, and reason about temporal information in clinical text.

Objective

To measure the accuracy of the TimeText for processing clinical discharge summaries.

Design

Six physicians with biomedical informatics training served as domain experts. Twenty discharge summaries were randomly selected for the evaluation. For each of the first 14 reports, 5 to 8 clinically important medical events were chosen. The temporal reasoning system generated temporal relations about the endpoints (start or finish) of pairs of medical events. Two experts (subjects) manually generated temporal relations for these medical events. The system and expert-generated results were assessed by four other experts (raters). All of the twenty discharge summaries were used to assess the system’s accuracy in answering time-oriented clinical questions. For each report, five to ten clinically plausible temporal questions about events were generated. Two experts generated answers to the questions to serve as the gold standard. We wrote queries to retrieve answers from system’s output.

Measurements

Correctness of generated temporal relations, recall of clinically important relations, and accuracy in answering temporal questions.

Results

The raters determined that 97% of subjects’ 295 generated temporal relations were correct and that 96.5% of the system’s 995 generated temporal relations were correct. The system captured 79% of 307 temporal relations determined to be clinically important by the subjects and raters. The system answered 84% of the temporal questions correctly.

Conclusion

The system encoded the majority of information identified by experts, and was able to answer simple temporal questions.     

Prognostic value of hedgehog signal component expressions in hepatoblastoma patients

Objective

Activation of hedgehog (Hh) pathway has been implicated in the development of human malignancies. Hh as well as related downstream target genes has been extensively studied in many kinds of malignant tumours for clinical diagnostic or prognostic utilities. This study aimed at investigating whether Hh molecules provides a molecular marker of hepatoblastoma malignancy.

Methods

We obtained tissue sections from 32 patients with hepatoblastoma as well as cholestasis and normal control. Immunohistochemical analysis were performed to determine Hh signal components in human hepatoblastoma. The prognostic significance of single expression of Hh signal components were evaluated using Cox proportional hazards regression models and Kaplan-Meier survival analysis for statistical analysis.

Results

Expression of Hh signal components showed an increase in hepatoblastoma compared with chole stasis and normal tissues. There was a positive correlation between Smo or Gli1 expression and tumor clinicopathological features, such as histological type, tumor grade, tumor size and clinical stage. Both Smo or Gli1 protein high expression was significantly associated with poor prognosis by univariate analyses and multivariate analyses.

Conclusions

Abnormal Hh signaling activation plays important roles in the malignant potential of hepatoblastoma. Gli1 expression is an independent prognostic marker.     

Using Wireless Handheld Computers to Seek Information at the Point of Care: An Evaluation by Clinicians

Objective

To evaluate: (1) the effectiveness of wireless handheld computers for online information retrieval in clinical settings; (2) the role of MEDLINE® in answering clinical questions raised at the point of care.

Design

A prospective single-cohort study: accompanying medical teams on teaching rounds, five internal medicine residents used and evaluated MD on Tap, an application for handheld computers, to seek answers in real time to clinical questions arising at the point of care.

Measurements

All transactions were stored by an intermediate server. Evaluators recorded clinical scenarios and questions, identified MEDLINE citations that answered the questions, and submitted daily and summative reports of their experience. A senior medical librarian corroborated the relevance of the selected citation to each scenario and question.

Results

Evaluators answered 68% of 363 background and foreground clinical questions during rounding sessions using a variety of MD on Tap features in an average session length of less than four minutes. The evaluator, the number and quality of query terms, the total number of citations found for a query, and the use of auto-spellcheck significantly contributed to the probability of query success.

Conclusion

Handheld computers with Internet access are useful tools for healthcare providers to access MEDLINE in real time. MEDLINE citations can answer specific clinical questions when several medical terms are used to form a query. The MD on Tap application is an effective interface to MEDLINE in clinical settings, allowing clinicians to quickly find relevant citations.     

From web search to healthcare utilization: privacy-sensitive studies from mobile data

Objective

We explore relationships between health information seeking activities and engagement with healthcare professionals via a privacy-sensitive analysis of geo-tagged data from mobile devices.

Materials and methods

We analyze logs of mobile interaction data stripped of individually identifiable information and location data. The data analyzed consist of time-stamped search queries and distances to medical care centers. We examine search activity that precedes the observation of salient evidence of healthcare utilization (EHU) (ie, data suggesting that the searcher is using healthcare resources), in our case taken as queries occurring at or near medical facilities.

Results

We show that the time between symptom searches and observation of salient evidence of seeking healthcare utilization depends on the acuity of symptoms. We construct statistical models that make predictions of forthcoming EHU based on observations about the current search session, prior medical search activities, and prior EHU. The predictive accuracy of the models varies (65%–90%) depending on the features used and the timeframe of the analysis, which we explore via a sensitivity analysis.

Discussion

We provide a privacy-sensitive analysis that can be used to generate insights about the pursuit of health information and healthcare. The findings demonstrate how large-scale studies of mobile devices can provide insights on how concerns about symptomatology lead to the pursuit of professional care.

Conclusion

We present new methods for the analysis of mobile logs and describe a study that provides evidence about how people transition from mobile searches on symptoms and diseases to the pursuit of healthcare in the world.     

Performance factors of mobile rich media job aids for community health workers

Objective

To study and analyze the possible benefits on performance of community health workers using point-of-care clinical guidelines implemented as interactive rich media job aids on small-format mobile platforms.

Design

A crossover study with one intervention (rich media job aids) and one control (traditional job aids), two periods, with 50 community health workers, each subject solving a total 15 standardized cases per period per period (30 cases in total per subject).

Measurements

Error rate per case and task, protocol compliance.

Results

A total of 1394 cases were evaluated. Intervention reduces errors by an average of 33.15% (p=0.001) and increases protocol compliance 30.18% (p<0.001).

Limitations

Medical cases were presented on human patient simulators in a laboratory setting, not on real patients.

Conclusion

These results indicate encouraging prospects for mHealth technologies in general, and the use of rich media clinical guidelines on cell phones in particular, for the improvement of community health worker performance in developing countries.     

Evaluation of computer-generated reminders to improve CD4 laboratory monitoring in sub-Saharan Africa: a prospective comparative study

Objective

Little evidence exists on effective interventions to integrate HIV-care guidelines into practices within developing countries. This study tested the hypothesis that clinical summaries with computer-generated reminders could improve clinicians'' compliance with CD4 testing guidelines in the resource-limited setting of sub-Saharan Africa.

Design

A prospective comparative study of two randomly selected outpatient adult HIV clinics in western Kenya. Printed summaries with reminders for overdue CD4 tests were made available to clinicians in the intervention clinic but not in the control clinic.

Measurements

Changes in order rates for overdue CD4 tests were compared between and within the two clinics.

Results

The computerized reminder system identified 717 encounters (21%) with overdue CD4 tests. Analysis by study assignment (regardless of summaries being printed or not) revealed that with computer-generated reminders, CD4 order rates were significantly higher in the intervention clinic compared to the control clinic (53% vs 38%, OR=1.80, CI 1.34 to 2.42, p<0.0001). When comparison was restricted to encounters where summaries with reminders were printed, order rates in intervention clinic were even higher (63%). The intervention clinic increased CD4 ordering from 42% before reminders to 63% with reminders (50% increase, OR=2.32, CI 1.67 to 3.22, p<0.0001), compared to control clinic with only 8% increase from prestudy baseline (CI 0.83 to 1.46, p=0.51).

Limitations

Evaluation was conducted at two clinics in a single institution.

Conclusions

Clinical summaries with computer-generated reminders significantly improved clinician compliance with CD4 testing guidelines in the resource-limited setting of sub-Saharan Africa. This technology can have broad applicability to improve quality of HIV care in these settings.     

Knowledge-based Methods to Help Clinicians Find Answers in MEDLINE

Objectives

Large databases of published medical research can support clinical decision making by providing physicians with the best available evidence. The time required to obtain optimal results from these databases using traditional systems often makes accessing the databases impractical for clinicians. This article explores whether a hybrid approach of augmenting traditional information retrieval with knowledge-based methods facilitates finding practical clinical advice in the research literature.

Design

Three experimental systems were evaluated for their ability to find MEDLINE citations providing answers to clinical questions of different complexity. The systems (SemRep, Essie, and CQA-1.0), which rely on domain knowledge and semantic processing to varying extents, were evaluated separately and in combination. Fifteen therapy and prevention questions in three categories (general, intermediate, and specific questions) were searched. The first 10 citations retrieved by each system were randomized, anonymized, and evaluated on a three-point scale. The reasons for ratings were documented.

Measurements

Metrics evaluating the overall performance of a system (mean average precision, binary preference) and metrics evaluating the number of relevant documents in the first several presented to a physician were used.

Results

Scores (mean average precision = 0.57, binary preference = 0.71) for fusion of the retrieval results of the three systems are significantly (p < 0.01) better than those for any individual system. All three systems present three to four relevant citations in the first five for any question type.

Conclusion

The improvements in finding relevant MEDLINE citations due to knowledge-based processing show promise in assisting physicians to answer questions in clinical practice.     

A Day in the Life of PubMed: Analysis of a Typical Day’s Query Log

Objective

To characterize PubMed usage over a typical day and compare it to previous studies of user behavior on Web search engines.

Design

We performed a lexical and semantic analysis of 2,689,166 queries issued on PubMed over 24 consecutive hours on a typical day.

Measurements

We measured the number of queries, number of distinct users, queries per user, terms per query, common terms, Boolean operator use, common phrases, result set size, MeSH categories, used semantic measurements to group queries into sessions, and studied the addition and removal of terms from consecutive queries to gauge search strategies.

Results

The size of the result sets from a sample of queries showed a bimodal distribution, with peaks at approximately 3 and 100 results, suggesting that a large group of queries was tightly focused and another was broad. Like Web search engine sessions, most PubMed sessions consisted of a single query. However, PubMed queries contained more terms.

Conclusion

PubMed’s usage profile should be considered when educating users, building user interfaces, and developing future biomedical information retrieval systems.     

Towards spoken clinical-question answering: evaluating and adapting automatic speech-recognition systems for spoken clinical questions

Objective

To evaluate existing automatic speech-recognition (ASR) systems to measure their performance in interpreting spoken clinical questions and to adapt one ASR system to improve its performance on this task.

Design and measurements

The authors evaluated two well-known ASR systems on spoken clinical questions: Nuance Dragon (both generic and medical versions: Nuance Gen and Nuance Med) and the SRI Decipher (the generic version SRI Gen). The authors also explored language model adaptation using more than 4000 clinical questions to improve the SRI system''s performance, and profile training to improve the performance of the Nuance Med system. The authors reported the results with the NIST standard word error rate (WER) and further analyzed error patterns at the semantic level.

Results

Nuance Gen and Med systems resulted in a WER of 68.1% and 67.4% respectively. The SRI Gen system performed better, attaining a WER of 41.5%. After domain adaptation with a language model, the performance of the SRI system improved 36% to a final WER of 26.7%.

Conclusion

Without modification, two well-known ASR systems do not perform well in interpreting spoken clinical questions. With a simple domain adaptation, one of the ASR systems improved significantly on the clinical question task, indicating the importance of developing domain/genre-specific ASR systems.     

Is Relevance Relevant? User Relevance Ratings May Not Predict the Impact of Internet Search on Decision Outcomes

Objective

A common measure of Internet search engine effectiveness is its ability to find documents that a user perceives as ‘relevant’. This study sought to test whether user provided relevance ratings for documents retrieved by an Internet search engine correlate with the decision outcome after use of a search engine.

Design

227 university students were asked to answer four randomly assigned consumer health questions, then to conduct an Internet search on one of two randomly assigned search engines of different performance, and to again answer the question.

Measurements

Participants were asked to provide a relevance score for each document retrieved as well as a pre and post search answer to each question.

Results

User relevance rankings had little or no predictive power. Relevance rankings were unable to predict whether the user of a search engine could correctly answer a question after search and could not differentiate between two search engines with statistically different performance in the hands of users. Only when users had strong prior knowledge of the questions, and the decision task was of low complexity, did relevance appear to have modest predictive power.

Conclusions

User provided relevance rankings taken in isolation seem to be of limited to no value when designing a search engine that will be used in a general-purpose setting. Relevance rankings may have a place in situations in which experts provide rankings, and decision tasks are of complexity commensurate with the abilities of the raters. A more natural metric of search engine performance may be a user''s ability to accurately complete a task, as this removes the inherent subjectivity of relevance rankings, and provides a direct and repeatable outcome measure which directly correlates with the performance of the search technology in the hands of users.     

The SMART Platform: early experience enabling substitutable applications for electronic health records

Objective

The Substitutable Medical Applications, Reusable Technologies (SMART) Platforms project seeks to develop a health information technology platform with substitutable applications (apps) constructed around core services. The authors believe this is a promising approach to driving down healthcare costs, supporting standards evolution, accommodating differences in care workflow, fostering competition in the market, and accelerating innovation.

Materials and methods

The Office of the National Coordinator for Health Information Technology, through the Strategic Health IT Advanced Research Projects (SHARP) Program, funds the project. The SMART team has focused on enabling the property of substitutability through an app programming interface leveraging web standards, presenting predictable data payloads, and abstracting away many details of enterprise health information technology systems. Containers—health information technology systems, such as electronic health records (EHR), personally controlled health records, and health information exchanges that use the SMART app programming interface or a portion of it—marshal data sources and present data simply, reliably, and consistently to apps.

Results

The SMART team has completed the first phase of the project (a) defining an app programming interface, (b) developing containers, and (c) producing a set of charter apps that showcase the system capabilities. A focal point of this phase was the SMART Apps Challenge, publicized by the White House, using http://www.challenge.gov website, and generating 15 app submissions with diverse functionality.

Conclusion

Key strategic decisions must be made about the most effective market for further disseminating SMART: existing market-leading EHR vendors, new entrants into the EHR market, or other stakeholders such as health information exchanges.     

Effectiveness of Topic-specific Infobuttons: A Randomized Controlled Trial

Objective

Infobuttons are decision support tools that provide links within electronic medical record systems to relevant content in online information resources. The aim of infobuttons is to help clinicians promptly meet their information needs. The objective of this study was to determine whether infobutton links that direct to specific content topics (“topic links”) are more effective than links that point to general overview content (“nonspecific links”).

Design

Randomized controlled trial with a control and an intervention group. Clinicians in the control group had access to nonspecific links, while those in the intervention group had access to topic links.

Measurements

Infobutton session duration, number of infobutton sessions, session success rate, and the self-reported impact that the infobutton session produced on decision making.

Results

The analysis was performed on 90 subjects and 3,729 infobutton sessions. Subjects in the intervention group spent 17.4% less time seeking for information (35.5 seconds vs. 43 seconds, p = 0.008) than those in the control group. Subjects in the intervention group used infobuttons 20.5% (22 sessions vs. 17.5 sessions, p = 0.21) more often than in the control group, but the difference was not significant. The information seeking success rate was equally high in both groups (89.4% control vs. 87.2% intervention, p = 0.99). Subjects reported a high positive clinical impact (i.e., decision enhancement or knowledge update) in 62% of the sessions.

Limitations

The exclusion of users with a low frequency of infobutton use and the focus on medication-related information needs may limit the generalization of the results. The session outcomes measurement was based on clinicians'' self-assessment and therefore prone to bias.

Conclusion

The results support the hypothesis that topic links are more efficient than nonspecific links regarding the time seeking for information. It is unclear whether the statistical difference demonstrated will result in a clinically significant impact. However, the overall results confirm previous evidence that infobuttons are effective at helping clinicians to answer questions at the point of care and demonstrate a modest incremental change in the efficiency of information delivery for routine users of this tool.     

Adoption of a wiki within a large internal medicine residency program: a 3-year experience

Objective

To describe the creation and evaluate the use of a wiki by medical residents, and to determine if a wiki would be a useful tool for improving the experience, efficiency, and education of housestaff.

Materials and methods

In 2008, a team of medical residents built a wiki containing institutional knowledge and reference information using Microsoft SharePoint. We tracked visit data for 3 years, and performed an audit of page views and updates in the second year. We evaluated the attitudes of medical residents toward the wiki using a survey.

Results

Users accessed the wiki 23 218, 35 094, and 40 545 times in each of three successive academic years from 2008 to 2011. In the year two audit, 85 users made a total of 1082 updates to 176 pages and of these, 91 were new page creations by 17 users. Forty-eight percent of residents edited a page. All housestaff felt the wiki improved their ability to complete tasks, and 90%, 89%, and 57% reported that the wiki improved their experience, efficiency, and education, respectively, when surveyed in academic year 2009–2010.

Discussion

A wiki is a useful and popular tool for organizing administrative and educational content for residents. Housestaff felt strongly that the wiki improved their workflow, but a smaller educational impact was observed. Nearly half of the housestaff edited the wiki, suggesting broad buy-in among the residents.

Conclusion

A wiki is a feasible and useful tool for improving information retrieval for house officers.     

Openness of patients' reporting with use of electronic records: psychiatric clinicians' views

Objectives

Improvements in electronic health record (EHR) system development will require an understanding of psychiatric clinicians'' views on EHR system acceptability, including effects on psychotherapy communications, data-recording behaviors, data accessibility versus security and privacy, data quality and clarity, communications with medical colleagues, and stigma.

Design

Multidisciplinary development of a survey instrument targeting psychiatric clinicians who recently switched to EHR system use, focus group testing, data analysis, and data reliability testing.

Measurements

Survey of 120 university-based, outpatient mental health clinicians, with 56 (47%) responding, conducted 18 months after transition from a paper to an EHR system.

Results

Factor analysis gave nine item groupings that overlapped strongly with five a priori domains. Respondents both praised and criticized the EHR system. A strong majority (81%) felt that open therapeutic communications were preserved. Regarding data quality, content, and privacy, clinicians (63%) were less willing to record highly confidential information and disagreed (83%) with including their own psychiatric records among routinely accessed EHR systems.

Limitations

single time point; single academic medical center clinic setting; modest sample size; lack of prior instrument validation; survey conducted in 2005.

Conclusions

In an academic medical center clinic, the presence of electronic records was not seen as a dramatic impediment to therapeutic communications. Concerns regarding privacy and data security were significant, and may contribute to reluctances to adopt electronic records in other settings. Further study of clinicians'' views and use patterns may be helpful in guiding development and deployment of electronic records systems.