Automatically Detecting Failures in Natural Language Processing Tools for Online Community Text

Background

The prevalence and value of patient-generated health text are increasing, but processing such text remains problematic. Although existing biomedical natural language processing (NLP) tools are appealing, most were developed to process clinician- or researcher-generated text, such as clinical notes or journal articles. In addition to being constructed for different types of text, other challenges of using existing NLP include constantly changing technologies, source vocabularies, and characteristics of text. These continuously evolving challenges warrant the need for applying low-cost systematic assessment. However, the primarily accepted evaluation method in NLP, manual annotation, requires tremendous effort and time.

Objective

The primary objective of this study is to explore an alternative approach—using low-cost, automated methods to detect failures (eg, incorrect boundaries, missed terms, mismapped concepts) when processing patient-generated text with existing biomedical NLP tools. We first characterize common failures that NLP tools can make in processing online community text. We then demonstrate the feasibility of our automated approach in detecting these common failures using one of the most popular biomedical NLP tools, MetaMap.

Methods

Using 9657 posts from an online cancer community, we explored our automated failure detection approach in two steps: (1) to characterize the failure types, we first manually reviewed MetaMap’s commonly occurring failures, grouped the inaccurate mappings into failure types, and then identified causes of the failures through iterative rounds of manual review using open coding, and (2) to automatically detect these failure types, we then explored combinations of existing NLP techniques and dictionary-based matching for each failure cause. Finally, we manually evaluated the automatically detected failures.

Results

From our manual review, we characterized three types of failure: (1) boundary failures, (2) missed term failures, and (3) word ambiguity failures. Within these three failure types, we discovered 12 causes of inaccurate mappings of concepts. We used automated methods to detect almost half of 383,572 MetaMap’s mappings as problematic. Word sense ambiguity failure was the most widely occurring, comprising 82.22% of failures. Boundary failure was the second most frequent, amounting to 15.90% of failures, while missed term failures were the least common, making up 1.88% of failures. The automated failure detection achieved precision, recall, accuracy, and F1 score of 83.00%, 92.57%, 88.17%, and 87.52%, respectively.

Conclusions

We illustrate the challenges of processing patient-generated online health community text and characterize failures of NLP tools on this patient-generated health text, demonstrating the feasibility of our low-cost approach to automatically detect those failures. Our approach shows the potential for scalable and effective solutions to automatically assess the constantly evolving NLP tools and source vocabularies to process patient-generated text.     

Telomere dysfunction accurately predicts clinical outcome in chronic lymphocytic leukaemia,even in patients with early stage disease

Defining the prognosis of individual cancer sufferers remains a significant clinical challenge. Here we assessed the ability of high‐resolution single telomere length analysis (STELA), combined with an experimentally derived definition of telomere dysfunction, to predict the clinical outcome of patients with chronic lymphocytic leukaemia (CLL). We defined the upper telomere length threshold at which telomere fusions occur and then used the mean of the telomere ‘fusogenic’ range as a prognostic tool. Patients with telomeres within the fusogenic range had a significantly shorter overall survival (P < 0·0001; Hazard ratio [HR] = 13·2, 95% confidence interval [CI] = 11·6–106·4) and this was preserved in early‐stage disease patients (P < 0·0001, HR=19·3, 95% CI = 17·8–802·5). Indeed, our assay allowed the accurate stratification of Binet stage A patients into those with indolent disease (91% survival at 10 years) and those with poor prognosis (13% survival at 10 years). Furthermore, patients with telomeres above the fusogenic mean showed superior prognosis regardless of their IGHV mutation status or cytogenetic risk group. In keeping with this finding, telomere dysfunction was the dominant variable in multivariate analysis. Taken together, this study provides compelling evidence for the use of high‐resolution telomere length analysis coupled with a definition of telomere dysfunction in the prognostic assessment of CLL.     

Spatiotemporal distribution of cortical processing of first and second languages in bilinguals. I. Effects of proficiency and linguistic setting

The study determined how spatiotemporal distribution of cortical activity to words in first and second language is affected by language, proficiency, and linguistic setting. Ten early bilinguals and 14 late adult bilinguals listened to pairs of words presented in Arabic (L1), Hebrew (L2), or in mixed pairs and indicated whether both words had the same meaning or not. Source current densities of event‐related potentials were estimated. Activity to first words in the pair lateralized to right hemisphere, higher to L1 than L2 during early processing (<300 ms) among both groups but only among late bilinguals during late processing (>300 ms). During early and late processing, activities were larger in mixed than monolinguistic settings among early bilinguals but lower in mixed than in monolinguistic settings among late bilinguals. Late processing in auditory regions was of larger magnitude in left than right hemispheres among both groups. Activity to second words in the pair was larger in mixed than in monolinguistic settings during both early and late processing among both groups. Early processing of second words in auditory regions lateralized to the right among early bilinguals and to the left among late bilinguals, whereas late processing did not differ between groups. Wernicke's area activity during late processing of L2 was larger on the right, while on the left no significant differences between languages were found. The results show that cortical language processing in bilinguals differs between early and late processing and these differences are modulated by linguistic proficiency and setting. Hum Brain Mapp 34:2863–2881, 2013. © 2012 Wiley Periodicals, Inc.     

Plasma levels of prorenin and Renin in blacks and whites: their relative abundance and associations with plasma aldosterone concentration

BackgroundAll renin arises from prorenin. The proportion of renin relative to prorenin could influence overall renin-angiotensin-aldosterone activity. We sought to determine whether prorenin levels were related to extracellular volume, as reflected by the levels of plasma renin activity (PRA), and to aldosterone.MethodsWe analyzed plasma levels of prorenin, renin, and aldosterone, as well as their interactions, in 129 young blacks and whites.ResultsBlacks had lower plasma renin concentration (PRC) and PRA, but had prorenin levels similar to whites (69 pg/ml in blacks vs. 62 pg/ml in whites, P = 0.41). As a result, the renin-to-total renin ratio was significantly lower in blacks (11.5% in blacks as compared to 19.8% in whites; P = 0.0001). Because prorenin also resides in tissues including the adrenal where it can bind to a specific receptor to generate angiotensin II, we examined the relationship of prorenin levels to plasma aldosterone concentrations (PAC). While a positive association between PRC and PAC was found in both blacks and whites, PAC was positively related to prorenin in whites (P = 0.04) but negatively in blacks, an observation that we hypothesize was due to reduced prorenin-to-renin conversion in blacks.ConclusionsWe observed a disproportionately high level of prorenin in blacks. These high circulating prorenin levels however do not result in greater adrenal angiotensin II and aldosterone production in healthy young blacks.American Journal of Hypertension 2012; doi:10.1038/ajh.2012.83.     

Anger as a predictor of psychological distress and self-harm ideation in inmates: A structured self-assessment diary study

Suicidal ideation and behaviour are common among inmates. Anger is found at exaggerated levels and has been associated with suicidal ideation and behaviour in inmate samples suggesting its possible salience in the prediction of suicide. The study investigated relationships between anger, psychological distress, and self-harm/suicidal ideation among inmates. The principles of Ecological Momentary Assessment were considered and a structured self-assessment diary was utilised to examine relationships between the variables of interest. Participants completed a structured self-assessment diary for six consecutive days which included momentary ratings of items describing psychological states of concurrent affects, thoughts, and appraisals related to anger, psychological distress, and self-harm/suicidal ideation. Psychometric assessment measures were also conducted. Temporal associations between predictors and outcomes were investigated. Multilevel modelling analyses were performed. Increased anger was significantly associated with concurrent high levels of self-harm ideation in inmates, when controlling for depression and hopelessness. Temporal analyses also revealed that anger at one time point did not predict suicidal ideation at the next time point. Elucidating the temporal nature of the relationship between anger, psychological distress, and self-harm/suicidal ideation has advanced understanding of the mechanisms of suicidal behaviour, by demonstrating an increased risk of suicide when a male inmate is angry.