Rapid Extraction of the Spatial Distribution of Physical Saliency and Semantic Informativeness from Natural Scenes in the Human Brain

Physically salient objects are thought to attract attention in natural scenes. However, research has shown that meaning maps, which capture the spatial distribution of semantically informative scene features, trump physical saliency in predicting the pattern of eye moments in natural scene viewing. Meaning maps even predict the fastest eye movements, suggesting that the brain extracts the spatial distribution of potentially meaningful scene regions very rapidly. To test this hypothesis, we applied representational similarity analysis to ERP data. The ERPs were obtained from human participants (N = 32, male and female) who viewed a series of 50 different natural scenes while performing a modified 1-back task. For each scene, we obtained a physical saliency map from a computational model and a meaning map from crowd-sourced ratings. We then used representational similarity analysis to assess the extent to which the representational geometry of physical saliency maps and meaning maps can predict the representational geometry of the neural response (the ERP scalp distribution) at each moment in time following scene onset. We found that a link between physical saliency and the ERPs emerged first (∼78 ms after stimulus onset), with a link to semantic informativeness emerging soon afterward (∼87 ms after stimulus onset). These findings are in line with previous evidence indicating that saliency is computed rapidly, while also indicating that information related to the spatial distribution of semantically informative scene elements is computed shortly thereafter, early enough to potentially exert an influence on eye movements.SIGNIFICANCE STATEMENT Attention may be attracted by physically salient objects, such as flashing lights, but humans must also be able to direct their attention to meaningful parts of scenes. Understanding how we direct attention to meaningful scene regions will be important for developing treatments for disorders of attention and for designing roadways, cockpits, and computer user interfaces. Information about saliency appears to be extracted rapidly by the brain, but little is known about the mechanisms that determine the locations of meaningful information. To address this gap, we showed people photographs of real-world scenes and measured brain activity. We found that information related to the locations of meaningful scene elements was extracted rapidly, shortly after the emergence of saliency-related information.     

From the Cover: Orientation saliency without visual cortex and target selection in archer fish

Our visual attention is attracted by salient stimuli in our environment and affected by primitive features such as orientation, color, and motion. Perceptual saliency due to orientation contrast has been extensively demonstrated in behavioral experiments with humans and other primates and is believed to be facilitated by the functional organization of the primary visual cortex. In behavioral experiments with the archer fish, a proficient hunter with remarkable visual abilities, we found an orientation saliency effect similar to that observed in human subjects. Given the enormous evolutionary distance between humans and archer fish, our findings suggest that orientation-based saliency constitutes a fundamental building block for efficient visual information processing.     

Parietal cortex integrates contextual and saliency signals during the encoding of natural scenes in working memory

The Brief presentation of a complex scene entails that only a few objects can be selected, processed indepth, and stored in memory. Both low‐level sensory salience and high‐level context‐related factors (e.g., the conceptual match/mismatch between objects and scene context) contribute to this selection process, but how the interplay between these factors affects memory encoding is largely unexplored. Here, during fMRI we presented participants with pictures of everyday scenes. After a short retention interval, participants judged the position of a target object extracted from the initial scene. The target object could be either congruent or incongruent with the context of the scene, and could be located in a region of the image with maximal or minimal salience. Behaviourally, we found a reduced impact of saliency on visuospatial working memory performance when the target was out‐of‐context. Encoding‐related fMRI results showed that context–congruent targets activated dorsoparietal regions, while context–incongruent targets de‐activated the ventroparietal cortex. Saliency modulated activity both in dorsal and ventral regions, with larger context‐related effects for salient targets. These findings demonstrate the joint contribution of knowledge‐based and saliency‐driven attention for memory encoding, highlighting a dissociation between dorsal and ventral parietal regions. Hum Brain Mapp 36:5003–5017, 2015. © 2015 Wiley Periodicals, Inc .     

Imaging the effects of methylphenidate on brain dopamine: new model on its therapeutic actions for attention-deficit/hyperactivity disorder.

Methylphenidate hydrochloride (MP) is an effective treatment for attention-deficit/hyperactivity disorder (ADHD), a common neurobehavioral disorder of childhood onset characterized by inattention, hyperactivity, and distractibility. Methylphenidate hydrochloride blocks the dopamine transporters (DAT), the main mechanism for removing dopamine (DA) from the synapse, is believed to be involved in its therapeutic properties. However, the mechanism(s) by which increases in DA improve symptomatology in ADHD are not completely understood. Our studies of the dopaminergic effects of MP in the human brain using positron emission tomography (PET) have shown that MP blocks DAT, and that extracellular DA increases in proportion to the level of blockade and the rate of DA release (modulated by DA cell firing). These DA increases are greater when MP is given concomitantly with a salient stimulus than with a neutral stimulus, documenting the context dependency of MP effects. Additionally, MP-induced increases in DA are associated with an enhanced perception of the stimulus as salient. We postulate the MP's therapeutic effects are due in part to its ability to enhance the magnitude of DA increases induced by stimuli that by themselves generate weak responses, enhancing their saliency and the attention and interest they elicit. We postulate that these effects would improve school performance.     

The influence of visual saliency on fixation patterns in individuals with Autism Spectrum Disorders

It is widely reported that individuals with Autism Spectrum Disorders (ASD) direct their attention in an atypical manner. When viewing complex scenes, typically developing individuals look at social aspects of scenes more rapidly than individuals with ASD. In the absence of a strong drive to extract social information, is something else capturing attention in these initial fixations, such as visually salient features? Twenty four high-functioning adolescents with ASD and 24 typically developing matched control participants viewed a series of indoor and outdoor scenes while their eye movements were tracked. Participants in both groups were more likely to fixate on salient regions in the first five fixations than later in viewing. Peak saliency at fixation occurred at fixation two for the typically developing participants but at fixation three for ASD participants. This difference was driven by typically developing participants looking at heads earlier than ASD participants - which are often visually salient. No differences between groups were observed for images in which the heads were not salient. We can therefore conclude that visual saliency impacts fixation location in a similar manner in individuals with ASD and those with typical development. It was found that social features in scenes (heads) captured attention much more than visually salient features, even in individuals with ASD.     

Social dimension and complexity differentially influence brain responses during feedback processing

Recent research emphasizes the importance of social factors during performance monitoring. Thus, the current study investigated the impact of social stimuli -such as communicative gestures- on feedback processing. Moreover, it addressed a shortcoming of previous studies, which failed to consider stimulus complexity as potential confounding factor. Twenty-four volunteers performed a time estimation task while their electroencephalogram was recorded. Either social complex, social non-complex, non-social complex, or non-social non-complex stimuli were used to provide performance feedback. No effects of social dimension or complexity were found for task performance. In contrast, Feedback-Related Negativity (FRN) and P300 amplitudes were sensitive to both factors, with larger FRN and P300 amplitudes after social compared to non-social stimuli, and larger FRN amplitudes after complex positive than non-complex positive stimuli. P2 amplitudes were solely sensitive to feedback valence and social dimension. Subjectively, social complex stimuli were rated as more motivating than non-social complex ones. Independently of each other, social dimension and visual complexity influenced amplitude variation during performance monitoring. Social stimuli seem to be perceived as more salient, which is corroborated by P2, FRN and P300 results, as well as by subjective ratings. This could be explained due to their given relevance during every day social interactions.     

Influence of ventral tegmental area input on cortico‐subcortical networks underlying action control and decision making

It is argued that the mesolimbic system has a more general function in processing all salient events, including and extending beyond rewards. Saliency was defined as an event that is unexpected due to its frequency of occurrence and elicits an attentional‐behavioral switch. Using functional magnetic resonance imaging (fMRI), signals were measured in response to the modulation of salience of rewarding and nonrewarding events during a reward‐based decision making task, the so called desire‐reason dilemma paradigm (DRD). Replicating previous findings, both frequent and infrequent, and therefore salient, reward stimuli elicited reliable activation of the ventral tegmental area (VTA) and ventral striatum (vStr). When immediate reward desiring contradicted the superordinate task‐goal, we found an increased activation of the VTA and vStr when the salient reward stimuli were presented compared to the nonsalient reward stimuli, indicating a boosting of activation in these brain regions. Furthermore, we found a significantly increased functional connectivity between the VTA and vStr, confirming the boosting of vStr activation via VTA input. Moreover, saliency per se without a reward association led to an increased activation of brain regions in the mesolimbic reward system as well as the orbitofrontal cortex (OFC), inferior frontal gyrus (IFG), and anterior cingulate cortex (ACC). Finally, findings uncovered multiple increased functional interactions between cortical saliency‐processing brain areas and the VTA and vStr underlying detection and processing of salient events and adaptive decision making.     

用于复杂场景背景减除的拮抗方向能量时空显著性

本文基于生物视觉启发构建了一种用于复杂场景下的无监督时空显著性检测方法。为了紧致描述场景局部动态纹理运动,首先提出分布式拮抗方向能量导出的动态纹理描述子获取局部运动信息。然后,本文探索了利用该表达结合扩展的自相似显著性框架得到时空显著性检测器。在多个极端动态背景和摄像头运动的图像序列中,利用本文提出的算法进行了背景减除实验。广泛的定性与定量实验结果表明本文方法快速且高效。在检测精度与计算代价平衡性方面,本文算法与最先进算法具有一定竞争力。     

Optimal reward harvesting in complex perceptual environments

The ability to choose rapidly among multiple targets embedded in a complex perceptual environment is key to survival. Targets may differ in their reward value as well as in their low-level perceptual properties (e.g., visual saliency). Previous studies investigated separately the impact of either value or saliency on choice; thus, it is not known how the brain combines these two variables during decision making. We addressed this question with three experiments in which human subjects attempted to maximize their monetary earnings by rapidly choosing items from a brief display. Each display contained several worthless items (distractors) as well as two targets, whose value and saliency were varied systematically. We compared the behavioral data with the predictions of three computational models assuming that (i) subjects seek the most valuable item in the display, (ii) subjects seek the most easily detectable item, and (iii) subjects behave as an ideal Bayesian observer who combines both factors to maximize the expected reward within each trial. Regardless of the type of motor response used to express the choices, we find that decisions are influenced by both value and feature-contrast in a way that is consistent with the ideal Bayesian observer, even when the targets’ feature-contrast is varied unpredictably between trials. This suggests that individuals are able to harvest rewards optimally and dynamically under time pressure while seeking multiple targets embedded in perceptual clutter.     

Processing scene context: fast categorization and object interference

The extent to which object identification is influenced by the background of the scene is still controversial. On the one hand, the global context of a scene might be considered as an ultimate representation, suggesting that object processing is performed almost systematically before scene context analysis. Alternatively, the gist of a scene could be extracted sufficiently early to be able to influence object categorization. It is thus essential to assess the processing time of scene context. In the present study, we used a go/no-go rapid visual categorization task in which subjects had to respond as fast as possible when they saw a "man-made environment", or a "natural environment", that was flashed for only 26 ms. "Man-made" and "natural" scenes were categorized with very high accuracy (both around 96%) and very short reaction times (median RT both around 390 ms). Compared with previous results from our group, these data demonstrate that global context categorization is remarkably fast: (1) it is as fast as object categorization [Fabre-Thorpe, M., Delorme, A., Marlot, C., & Thorpe, S. (2001). A limit to the speed of processing in ultra-rapid visual categorization of novel natural scenes. Journal of Cognitive Neuroscience, 13(2), 171-180]; (2) it is faster than contextual categorization at more detailed levels such as sea, mountain, indoor or urban contexts [Rousselet, G. A., Joubert, O. R., & Fabre-Thorpe, M. (2005). How long to get to the "gist" of real-world natural scenes? Visual Cognition, 12(6), 852-877]. Further analysis showed that the efficiency of contextual categorization was impaired by the presence of a salient object in the scene especially when the object was incongruent with the context. Processing of natural scenes might thus involve in parallel the extraction of the global gist of the scene and the concurrent object processing leading to categorization. These data also suggest early interactions between scene and object representations compatible with contextual influences on object categorization in a parallel network.