HANA: Hierarchical Attention Network Assembling for Semantic Segmentation

Semantic segmentation is a crucial issue in the field of computer vision, and it aims to assign each pixel in an image to a semantic object category. Modern cognitive research has presented that the biological system contains hidden features and explicit features, although they both contain useful information, the hidden features need further processing to make them explicit or clear. Inspired by this theory, a semantic segmentation framework named hierarchical attention network assembling is proposed. Multiple auxilary information of different levels corresponding to the two kinds of features of the cognitive system are exploited. Then we further process the hidden information to make them explicit for the semantic segmentation. While in the traditional methods, limited assistance of the auxiliary tasks with only hidden information is provided. In this study, the attention mechanism is utilized and two auxiliary tasks are introduced as attention modules to give explicit guidance to the semantic segmentation task. Two hierarchical sub-networks—an object-level bounding box attention network and an edge-level boundary attention network together serve as explicit auxiliary tasks, of which the first network driven by the object detection aims to aggrandize the consistency constraint of pixels belonging to the same object, and the second one driven by the boundary detection aims to improve the segmentation accuracy within the boundary regions. With the proposed method, the performance achieves 78.3% mean IOU on PASCAL VOC 2012. The explicit guidance of the two auxiliary tasks can well assist the semantic segmentation task.

     

Visual cues in the interpretation of medical images

The use of the computer in imaging has provided great versatility for the display of medical image information. In order to be useful in a medical sense, the display must be perceptually acceptable to the human observer, who must be able to extract the relevant diagnostic information from the image. Diagnostic information is defined by the imaging task, and the concept of task-dependent image quality is, therefore, very important. Visualization and estimation tasks may require different visual cues for their performance. Visualization tasks require strong boundary cues, whereas intensity estimation tasks require texture cues. Both tasks can be aided by appropriate image displays. Color can be helpful in estimation tasks, and three-dimensional display can aid visualization tasks. No matter what the task or the image, as long as an observer is needed to read out the information, performance is the final arbiter of the goodness of the image.     

Selecting salient objects in real scenes: An oscillatory correlation model

Attention is a critical mechanism for visual scene analysis. By means of attention, it is possible to break down the analysis of a complex scene to the analysis of its parts through a selection process. Empirical studies demonstrate that attentional selection is conducted on visual objects as a whole. We present a neurocomputational model of object-based selection in the framework of oscillatory correlation. By segmenting an input scene and integrating the segments with their conspicuity obtained from a saliency map, the model selects salient objects rather than salient locations. The proposed system is composed of three modules: a saliency map providing saliency values of image locations, image segmentation for breaking the input scene into a set of objects, and object selection which allows one of the objects of the scene to be selected at a time. This object selection system has been applied to real gray-level and color images and the simulation results show the effectiveness of the system.     

The influence of visual saliency on fixation patterns in individuals with Autism Spectrum Disorders

It is widely reported that individuals with Autism Spectrum Disorders (ASD) direct their attention in an atypical manner. When viewing complex scenes, typically developing individuals look at social aspects of scenes more rapidly than individuals with ASD. In the absence of a strong drive to extract social information, is something else capturing attention in these initial fixations, such as visually salient features? Twenty four high-functioning adolescents with ASD and 24 typically developing matched control participants viewed a series of indoor and outdoor scenes while their eye movements were tracked. Participants in both groups were more likely to fixate on salient regions in the first five fixations than later in viewing. Peak saliency at fixation occurred at fixation two for the typically developing participants but at fixation three for ASD participants. This difference was driven by typically developing participants looking at heads earlier than ASD participants - which are often visually salient. No differences between groups were observed for images in which the heads were not salient. We can therefore conclude that visual saliency impacts fixation location in a similar manner in individuals with ASD and those with typical development. It was found that social features in scenes (heads) captured attention much more than visually salient features, even in individuals with ASD.     

Visual object and face processing in mild-to-moderate Alzheimer's disease: from segmentation to imagination

Little is known about the fate of higher level visual perception and visual mental imagery in the early stages of Alzheimer's disease (AD). In this study, we assessed these abilities in a group of mild-to-moderate AD patients using tasks selected to satisfy two main criteria. First, they have been shown to be sensitive to impairments of perception and imagery caused by other neurological conditions. Second, they test specific stages of visual perception and cognition in a reasonably selective manner. These stages were (in their normal order of occurrence during perception): the segmentation of different local points of the visual field into regions belonging to distinct objects; the representation of the shapes of these segmented regions in the image; the construction of more abstract shape representations that possess constancy over changes in size, location, orientation or illumination (assessed separately for faces and objects); the use of these perceived shape representations to access stored shape representations; and the access of lexical semantic representations from these high-level visual representations. Additional tasks tested the top-down activation of earlier visual representations from the semantic level in visual mental imagery. Our findings indicate small, but in most cases reliable, impairments in visual perception, which are independent of degree of cognitive decline. Deficits in basic shape processing influenced performance on some higher level visual tasks, but did not contribute to poor performance on face processing, or to the profound deficit on object naming. The latter of these is related to semantic-lexical impairment.     

On the difference between working memory and attentional set

Previous work has shown that distractors present in a visual search display attract attention when they match objects kept in visual working memory. It seems that maintaining an object in working memory is functionally identical to adopting an attentional set for that object. We test this conjecture by asking observers to perform a memory task as well as a visual search task (in which memory-related distractors could return), but to leave the observer uncertain as to which of these tasks would have to be completed first. This way, observers ought to more readily look for the memorized information, rather than just remember it. Memory-related distractor effects were larger than when participants knew the order of the tasks beforehand, consistent with the idea that trying to attend to something involves additional processes or representations beyond those needed for simply storing an item.     

Neural Basis of Visual Attentional Orienting in Childhood Autism Spectrum Disorders

We examined spontaneous attention orienting to visual salience in stimuli without social significance using a modified Dot-Probe task during functional magnetic resonance imaging in high-functioning preadolescent children with Autism Spectrum Disorder (ASD) and age- and IQ-matched control children. While the magnitude of attentional bias (faster response to probes in the location of solid color patch) to visually salient stimuli was similar in the groups, activation differences in frontal and temporoparietal regions suggested hyper-sensitivity to visual salience or to sameness in ASD children. Further, activation in a subset of those regions was associated with symptoms of restricted and repetitive behavior. Thus, atypicalities in response to visual properties of stimuli may drive attentional orienting problems associated with ASD.     

The role of multisensory memories in unisensory object discrimination

Past multisensory experiences can influence current unisensory processing and memory performance. Repeated images are better discriminated if initially presented as auditory-visual pairs, rather than only visually. An experience's context thus plays a role in how well repetitions of certain aspects are later recognized. Here, we investigated factors during the initial multisensory experience that are essential for generating improved memory performance. Subjects discriminated repeated versus initial image presentations intermixed within a continuous recognition task. Half of initial presentations were multisensory, and all repetitions were only visual. Experiment 1 examined whether purely episodic multisensory information suffices for enhancing later discrimination performance by pairing visual objects with either tones or vibrations. We could therefore also assess whether effects can be elicited with different sensory pairings. Experiment 2 examined semantic context by manipulating the congruence between auditory and visual object stimuli within blocks of trials. Relative to images only encountered visually, accuracy in discriminating image repetitions was significantly impaired by auditory-visual, yet unaffected by somatosensory-visual multisensory memory traces. By contrast, this accuracy was selectively enhanced for visual stimuli with semantically congruent multisensory pasts and unchanged for those with semantically incongruent multisensory pasts. The collective results reveal opposing effects of purely episodic versus semantic information from auditory-visual multisensory events. Nonetheless, both types of multisensory memory traces are accessible for processing incoming stimuli and indeed result in distinct visual object processing, leading to either impaired or enhanced performance relative to unisensory memory traces. We discuss these results as supporting a model of object-based multisensory interactions.     

Role of the anterior insular cortex in integrative causal signaling during multisensory auditory–visual attention

Coordinated attention to information from multiple senses is fundamental to our ability to respond to salient environmental events, yet little is known about brain network mechanisms that guide integration of information from multiple senses. Here we investigate dynamic causal mechanisms underlying multisensory auditory–visual attention, focusing on a network of right‐hemisphere frontal–cingulate–parietal regions implicated in a wide range of tasks involving attention and cognitive control. Participants performed three ‘oddball’ attention tasks involving auditory, visual and multisensory auditory–visual stimuli during fMRI scanning. We found that the right anterior insula (rAI) demonstrated the most significant causal influences on all other frontal–cingulate–parietal regions, serving as a major causal control hub during multisensory attention. Crucially, we then tested two competing models of the role of the rAI in multisensory attention: an ‘integrated’ signaling model in which the rAI generates a common multisensory control signal associated with simultaneous attention to auditory and visual oddball stimuli versus a ‘segregated’ signaling model in which the rAI generates two segregated and independent signals in each sensory modality. We found strong support for the integrated, rather than the segregated, signaling model. Furthermore, the strength of the integrated control signal from the rAI was most pronounced on the dorsal anterior cingulate and posterior parietal cortices, two key nodes of saliency and central executive networks respectively. These results were preserved with the addition of a superior temporal sulcus region involved in multisensory processing. Our study provides new insights into the dynamic causal mechanisms by which the AI facilitates multisensory attention.     

Object-based and action-based visual perception in children with spina bifida and hydrocephalus.

Children with spina bifida and hydrocephalus (SBH) have long been known to have difficulties with visual perception. We studied how children with SBH perform 12 visual perception tasks requiring object identification, multistable representations of visual space, or visually guided overt actions. Four tasks required object-based processing (visual constancy illusions, face recognition, recognition of fragmented objects, line orientation). Four tasks required the representation of visual space in egocentric coordinates (stereopsis, visual figure-ground identification, perception of multistable figures, egocentric mental rotation). Four tasks required the coupling of visual space to overt movement (visual pursuit, figure drawing, visually guided route finding, visually guided route planning). Effect sizes, measuring the magnitude of the difference between SBH children and controls, were consistently larger for action-based than object-based visual perception tasks. Within action-based tasks, effect sizes were large and roughly comparable for tasks requiring the representation of visual space and for tasks requiring visually guided action. The results are discussed in terms of the physical and brain problems of children with SBH that limit their ability to build effective situation models of space.     

Fixation and saliency during search of natural scenes: The case of visual agnosia

Models of eye movement control in natural scenes often distinguish between stimulus-driven processes (which guide the eyes to visually salient regions) and those based on task and object knowledge (which depend on expectations or identification of objects and scene gist). In the present investigation, the eye movements of a patient with visual agnosia were recorded while she searched for objects within photographs of natural scenes and compared to those made by students and age-matched controls. Agnosia is assumed to disrupt the top-down knowledge available in this task, and so may increase the reliance on bottom-up cues. The patient's deficit in object recognition was seen in poor search performance and inefficient scanning. The low-level saliency of target objects had an effect on responses in visual agnosia, and the most salient region in the scene was more likely to be fixated by the patient than by controls. An analysis of model-predicted saliency at fixation locations indicated a closer match between fixations and low-level saliency in agnosia than in controls. These findings are discussed in relation to saliency-map models and the balance between high and low-level factors in eye guidance.     

Complementary hemispheric asymmetries in object naming and recognition: A voxel-based correlational study

Cognitive neuroscientific research proposes complementary hemispheric asymmetries in naming and recognising visual objects, with a left temporal lobe advantage for object naming and a right temporal lobe advantage for object recognition. Specifically, it has been proposed that the left inferior temporal lobe plays a mediational role linking conceptual information with word forms and vice versa, while the right inferior temporal lobe supports the retrieval of conceptual knowledge from visual input. To test these hypotheses, we administered four behavioural tasks to fifteen patients with temporal lobe brain damage, and correlated their behavioural scores with voxel-based measures of neuronal integrity (signal intensities) in whole-brain analyses. The behavioural paradigms included four tasks. Two were verbal tasks: (a) picture naming requiring the mapping of conceptual knowledge to word forms, (b) semantic categorisation of words requiring the reverse mapping of word forms to conceptual knowledge, and two were visual object tasks with no verbal component, both of which required the retrieval of conceptual information from visual objects, i.e., (c) visual object categorisation and (d) normal and chimera object decisions. Performance on the verbal tasks correlated with the neural integrity of partially overlapping left inferior and anterior temporal lobe regions, while performance on the object tasks correlated with the neural integrity of overlapping regions in right inferior and anterior temporal lobe. These findings support the notion of complementary hemispheric advantages for object naming and recognition, and further suggest that the classical language model emphasising posterior regions in the mapping between word forms and conceptual knowledge should be extended to include left inferior temporal lobe.     

Emotion perception in schizophrenia: an eye movement study comparing the effectiveness of risperidone vs. haloperidol

We used a psychophysiological marker of visual attention (the visual scanpath) to investigate the effects of atypical (risperidone) vs. typical (haloperidol) antipsychotic medication on facial emotion perception in schizophrenia (n=28) and healthy control (n=28) groups. Of the schizophrenia subjects, 15 were prescribed risperidone. Visual scanpaths to 'happy', 'sad' and 'neutral' faces were recorded using video-oculography, and concurrent emotion-recognition accuracy was assessed using multiple-option tasks. Compared to control subjects, both schizophrenia subgroups showed a restriction in visual scanning (reduced total fixation number and decreased scanpath length). Haloperidol-treated schizophrenia subjects exhibited an additional and consistent pattern of reduced attention (fixation) to salient features for neutral and happy. By contrast, risperidone-treated subjects showed a relatively greater attention to salient features for these expressions, in which they did not differ from controls. Recognition accuracy for happy and neutral showed a similar lack of impairment. These findings suggest that risperidone may play a specific role in schizophrenia in the ability to attend to salient features, and to integrate this information into an accurate percept for neutral and positive expressions in particular.     

A Biologically Inspired Framework for Visual Information Processing and an Application on Modeling Bottom-Up Visual Attention

Background

An emerging trend in visual information processing is toward incorporating some interesting properties of the ventral stream in order to account for some limitations of machine learning algorithms. Selective attention and cortical magnification are two such important phenomena that have been the subject of a large body of research in recent years. In this paper, we focus on designing a new model for visual acquisition that takes these important properties into account.

Methods

We propose a new framework for visual information acquisition and representation that emulates the architecture of the primate visual system by integrating features such as retinal sampling and cortical magnification while avoiding spatial deformations and other side effects produced by models that tried to implement these two features. It also explicitly integrates the notion of visual angle, which is rarely taken into account by vision models. We argue that this framework can provide the infrastructure for implementing vision tasks such as object recognition and computational visual attention algorithms.

Results

To demonstrate the utility of the proposed vision framework, we propose an algorithm for bottom-up saliency prediction implemented using the proposed architecture. We evaluate the performance of the proposed model on the MIT saliency benchmark and show that it attains state-of-the-art performance, while providing some advantages over other models.

Conclusion

Here is a summary of the main contributions of this paper: (1) Introducing a new bio-inspired framework for visual information acquisition and representation that offers the following properties: (a) Providing a method for taking the distance between an image and the viewer into account. This is done by incorporating a visual angle parameter which is ignored by most visual acquisition models. (b) Reducing the amount of visual information acquired by introducing a new scheme for emulating retinal sampling and cortical magnification effects observed in the ventral stream. (2) Providing a concrete application of the proposed framework by using it as a substrate for building a new saliency-based visual attention model, which is shown to attain state-of-the-art performance on the MIT saliency benchmark. (3) Providing an online Git repository that implements the introduced framework that is meant to be developed as a scalable, collaborative project.

     

Astronomical image segmentation by self-organizing neural networks and wavelets.

Standard image segmentation methods may not be able to segment astronomical images because their special nature. We present an algorithm for astronomical image segmentation based on self-organizing neural networks and wavelets. We begin by performing wavelet decomposition of the image. The segmentation process has two steps. In the first we separate the stars and other prominent objects using the second plane (w(2)) of the wavelet decomposition, which has little noise but retains enough signal to represent those objects. This method was as least as effective as the traditional source extraction methods in isolating bright objects both from the background and from extended sources. In the second step the rest of the image (extended sources and background) is segmented using a self-organizing neural network. The result is a predetermined number of clusters, which we associate with extended regions plus a small region for each star or bright object. We have applied the algorithm to segment images of both galaxies and planets. The results show that the simultaneous use of all the scales in the self-organizing neural network helps the segmentation process, since it takes into account not only the intensity level, but also both the high and low frequencies present in the image. The connectivity of the regions obtained also shows that the algorithm is robust in the presence of noise. The method can also be applied to restored images.     

The pulvinar and visual salience.

One of the major problems of living in a rich visual environment is deciding which particular object or location should be chosen for complete processing or attention; that is, deciding which object is most salient at any particular time. The pulvinar has enlarged substantially during evolution, although little has previously been known about its function. Recent studies suggest that the pulvinar contains neurons that generate signals related to the salience of visual objects. This evidence includes: (1) anatomical and physiological observations of visual function; (2) augmented responses in the pulvinar for visual stimuli presented in important contexts; (3) suppression of activity for stimuli presented in irrelevant conditions; (4) thalamic modulation producing behavioral changes in cued attention paradigms; and (5) similar changes with visual distracter tasks.     

Oscillations during observations: Dynamic oscillatory networks serving visuospatial attention

The dynamic allocation of neural resources to discrete features within a visual scene enables us to react quickly and accurately to salient environmental circumstances. A network of bilateral cortical regions is known to subserve such visuospatial attention functions; however the oscillatory and functional connectivity dynamics of information coding within this network are not fully understood. Particularly, the coding of information within prototypical attention‐network hubs and the subsecond functional connections formed between these hubs have not been adequately characterized. Herein, we use the precise temporal resolution of magnetoencephalography (MEG) to define spectrally specific functional nodes and connections that underlie the deployment of attention in visual space. Twenty‐three healthy young adults completed a visuospatial discrimination task designed to elicit multispectral activity in visual cortex during MEG, and the resulting data were preprocessed and reconstructed in the time–frequency domain. Oscillatory responses were projected to the cortical surface using a beamformer, and time series were extracted from peak voxels to examine their temporal evolution. Dynamic functional connectivity was then computed between nodes within each frequency band of interest. We find that visual attention network nodes are defined functionally by oscillatory frequency, that the allocation of attention to the visual space dynamically modulates functional connectivity between these regions on a millisecond timescale, and that these modulations significantly correlate with performance on a spatial discrimination task. We conclude that functional hubs underlying visuospatial attention are segregated not only anatomically but also by oscillatory frequency, and importantly that these oscillatory signatures promote dynamic communication between these hubs. Hum Brain Mapp 38:5128–5140, 2017. © 2017 Wiley Periodicals, Inc.     

The inversion effect in visual word form processing

Reading is one of the best well-practiced visual tasks for modern people. We investigated how the visual cortex analyzes spatial configuration in written words by studying the inversion effect in Chinese character processing. We measured the psychometric functions and brain activations for upright real-characters and non-characters and their inverted (upside down) versions. In the psychophysical experiment, the real-characters showed an inversion effect at both 1° and 4° eccentricities, while the non-characters showed no inversion effect for all eccentricities tested. In the functional magnetic resonance image (fMRI) experiment, the left fusiform gyrus and a small area in the bilateral lateral occipital regions showed a significant differential activation between upright and inverted real-characters. The bilateral fusiform gyri also show differential activation between upright real- and non-characters. The dorsal lateral occipital regions showed character-selective activation when compared with scrambled lines. The result suggested that the occipitoparietal regions may analyze the local features of an object regardless of its familiarity. Therefore, the lateral occipital regions may play an intermediate role in integrating the local information in an object. Finally, the fusiform gyrus plays a critical role in analyzing global configurations of a visual word form. This is consistent with the notion that the human visual cortex analyzes an object in a hierarchical way.     

Stimulus specific adaptation in excited but not in inhibited cells in inferotemporal cortex of Macaque

Many cells in inferotemporal cortex respond more actively to a novel presentation than to a subsequent re-presentation of the same image, exhibiting stimulus specific adaptation (SSA). Previously, analysis of this adaptation was limited to visually excited cells, excluding visually inhibited cells. In the present experiment we studied 654 cells in four macaques performing visual tasks. Strong SSA (P < 0.0001) was observed in those cells which were excited by visual stimuli. This adaptation was also seen in the subset of such cells which, though excited by visual stimuli, failed to show visual specificity in their responses. Interestingly, no SSA (P > 0.1) was observed in the group of cells inhibited by visual stimuli. Furthermore, most inhibited cells failed to show visual specificity. This lack of visual specificity and SSA suggest that the visually inhibited cells have a limited role in the detailed information processing of visual perception and memory activated by the tasks used in the present experiments.     

Oculomotor evidence for neocortical systems but not cerebellar dysfunction in autism

OBJECTIVE: To investigate the functional integrity of cerebellar and frontal systems in autism using oculomotor paradigms. BACKGROUND: Cerebellar and neocortical systems models of autism have been proposed. Courchesne and colleagues have argued that cognitive deficits such as shifting attention disturbances result from dysfunction of vermal lobules VI and VII. Such a vermal deficit should be associated with dysmetric saccadic eye movements because of the major role these areas play in guiding the motor precision of saccades. In contrast, neocortical models of autism predict intact saccade metrics, but impairments on tasks requiring the higher cognitive control of saccades. METHODS: A total of 26 rigorously diagnosed nonmentally retarded autistic subjects and 26 matched healthy control subjects were assessed with a visually guided saccade task and two volitional saccade tasks, the oculomotor delayed-response task and the antisaccade task. RESULTS: Metrics and dynamics of the visually guided saccades were normal in autistic subjects, documenting the absence of disturbances in cerebellar vermal lobules VI and VII and in automatic shifts of visual attention. Deficits were demonstrated on both volitional saccade tasks, indicating dysfunction in the circuitry of prefrontal cortex and its connections with the parietal cortex, and associated cognitive impairments in spatial working memory and in the ability to voluntarily suppress context-inappropriate responses. CONCLUSIONS: These findings demonstrate intrinsic neocortical, not cerebellar, dysfunction in autism, and parallel deficits in higher order cognitive mechanisms and not in elementary attentional and sensorimotor systems in autism.