Neural correlates of disparity-defined shape discrimination in the human brain

Binocular disparity, the slight differences between the images registered by our two eyes, provides an important cue when estimating the three-dimensional (3D) structure of the complex environment we inhabit. Sensitivity to binocular disparity is evident at multiple levels of the visual hierarchy in the primate brain, from early visual cortex to parietal and temporal areas. However, the relationship between activity in these areas and key perceptual functions that exploit disparity information for 3D shape perception remains an important open question. Here we investigate the link between human cortical activity and the perception of disparity-defined shape, measuring fMRI responses concurrently with psychophysical shape judgments. We parametrically degraded the coherence of shapes by shuffling the spatial position of dots whose disparity defined the 3D structure and investigated the effect of this stimulus manipulation on both cortical activity and shape discrimination. We report significant relationships between shape coherence and fMRI response in both dorsal (V3, hMT+/V5) and ventral (LOC) visual areas that correspond to the observers' discrimination performance. In contrast to previous suggestions of a dichotomy of disparity-related processes in the ventral and dorsal streams, these findings are consistent with proposed interactions between these pathways that may mediate a continuum of processes important in perceiving 3D shape from coarse contour segmentation to fine curvature estimation.     

Object-selective responses in the human motion area MT/MST.

The perception of moving objects and our successful interaction with them entail that the visual system integrates shape and motion information about objects. However, neuroimaging studies have implicated different human brain regions in the analysis of visual motion (medial temporal cortex; MT/MST) and shape (lateral occipital complex; LOC), consistent with traditional approaches in visual processing that attribute shape and motion processing to anatomically and functionally separable neural mechanisms. Here we demonstrate object-selective fMRI responses (higher responses for intact than for scrambled images of objects) in MT/MST, and especially in a ventral subregion of MT/MST, suggesting that human brain regions involved mainly in the processing of visual motion are also engaged in the analysis of object shape.     

Remapping in human visual cortex

With each eye movement, stationary objects in the world change position on the retina, yet we perceive the world as stable. Spatial updating, or remapping, is one neural mechanism by which the brain compensates for shifts in the retinal image caused by voluntary eye movements. Remapping of a visual representation is believed to arise from a widespread neural circuit including parietal and frontal cortex. The current experiment tests the hypothesis that extrastriate visual areas in human cortex have access to remapped spatial information. We tested this hypothesis using functional magnetic resonance imaging (fMRI). We first identified the borders of several occipital lobe visual areas using standard retinotopic techniques. We then tested subjects while they performed a single-step saccade task analogous to the task used in neurophysiological studies in monkeys, and two conditions that control for visual and motor effects. We analyzed the fMRI time series data with a nonlinear, fully Bayesian hierarchical statistical model. We identified remapping as activity in the single-step task that could not be attributed to purely visual or oculomotor effects. The strength of remapping was roughly monotonic with position in the visual hierarchy: remapped responses were largest in areas V3A and hV4 and smallest in V1 and V2. These results demonstrate that updated visual representations are present in cortical areas that are directly linked to visual perception.     

Neural coding of global form in the human visual cortex

Extensive psychophysical and computational work proposes that the perception of coherent and meaningful structures in natural images relies on neural processes that convert information about local edges in primary visual cortex to complex object features represented in the temporal cortex. However, the neural basis of these mid-level vision mechanisms in the human brain remains largely unknown. Here, we examine functional MRI (fMRI) selectivity for global forms in the human visual pathways using sensitive multivariate analysis methods that take advantage of information across brain activation patterns. We use Glass patterns, parametrically varying the perceived global form (concentric, radial, translational) while ensuring that the local statistics remain similar. Our findings show a continuum of integration processes that convert selectivity for local signals (orientation, position) in early visual areas to selectivity for global form structure in higher occipitotemporal areas. Interestingly, higher occipitotemporal areas discern differences in global form structure rather than low-level stimulus properties with higher accuracy than early visual areas while relying on information from smaller but more selective neural populations (smaller voxel pattern size), consistent with global pooling mechanisms of local orientation signals. These findings suggest that the human visual system uses a code of increasing efficiency across stages of analysis that is critical for the successful detection and recognition of objects in complex environments.     

The integration of motion and disparity cues to depth in dorsal visual cortex

Humans exploit a range of visual depth cues to estimate three-dimensional structure. For example, the slant of a nearby tabletop can be judged by combining information from binocular disparity, texture and perspective. Behavioral tests show humans combine cues near-optimally, a feat that could depend on discriminating the outputs from cue-specific mechanisms or on fusing signals into a common representation. Although fusion is computationally attractive, it poses a substantial challenge, requiring the integration of quantitatively different signals. We used functional magnetic resonance imaging (fMRI) to provide evidence that dorsal visual area V3B/KO meets this challenge. Specifically, we found that fMRI responses are more discriminable when two cues (binocular disparity and relative motion) concurrently signal depth, and that information provided by one cue is diagnostic of depth indicated by the other. This suggests a cortical node important when perceiving depth, and highlights computations based on fusion in the dorsal stream.     

Decoding the visual and subjective contents of the human brain

The potential for human neuroimaging to read out the detailed contents of a person's mental state has yet to be fully explored. We investigated whether the perception of edge orientation, a fundamental visual feature, can be decoded from human brain activity measured with functional magnetic resonance imaging (fMRI). Using statistical algorithms to classify brain states, we found that ensemble fMRI signals in early visual areas could reliably predict on individual trials which of eight stimulus orientations the subject was seeing. Moreover, when subjects had to attend to one of two overlapping orthogonal gratings, feature-based attention strongly biased ensemble activity toward the attended orientation. These results demonstrate that fMRI activity patterns in early visual areas, including primary visual cortex (V1), contain detailed orientation information that can reliably predict subjective perception. Our approach provides a framework for the readout of fine-tuned representations in the human brain and their subjective contents.     

The role of temporo-parietal junction (TPJ) in global Gestalt perception

Grouping processes enable the coherent perception of our environment. A number of brain areas has been suggested to be involved in the integration of elements into objects including early and higher visual areas along the ventral visual pathway as well as motion-processing areas of the dorsal visual pathway. However, integration not only is required for the cortical representation of individual objects, but is also essential for the perception of more complex visual scenes consisting of several different objects and/or shapes. The present fMRI experiments aimed to address such integration processes. We investigated the neural correlates underlying the global Gestalt perception of hierarchically organized stimuli that allowed parametrical degrading of the object at the global level. The comparison of intact versus disturbed perception of the global Gestalt revealed a network of cortical areas including the temporo-parietal junction (TPJ), anterior cingulate cortex and the precuneus. The TPJ location corresponds well with the areas known to be typically lesioned in stroke patients with simultanagnosia following bilateral brain damage. These patients typically show a deficit in identifying the global Gestalt of a visual scene. Further, we found the closest relation between behavioral performance and fMRI activation for the TPJ. Our data thus argue for a significant role of the TPJ in human global Gestalt perception.     

Brain activation during dichoptic presentation of optic flow stimuli

The processing of optic flow fields in motion-sensitive areas in human visual cortex was studied with BOLD (blood oxygen level dependent) contrast in functional magnetic resonance imaging (fMRI). Subjects binocularly viewed optic flow fields in plane (monoptic) or in stereo depth (dichoptic) with various degrees of disparity and increasing radial speed. By varying the directional properties of the stimuli (expansion, spiral motion, random), we explored whether the BOLD effect reflected neuronal responses to these different forms of optic flow. The results suggest that BOLD contrast as assessed by fMRI methods reflects the neural processing of optic flow information in motion-sensitive cortical areas. Furthermore, small but replicable disparity-selective responses were found in parts of Brodmann's area 19.     

High-resolution imaging reveals highly selective nonface clusters in the fusiform face area

A region in ventral human cortex (fusiform face area, FFA) thought to be important for face perception responds strongly to faces and less strongly to nonface objects. This pattern of response may reflect a uniform face-selective neural population or activity averaged across populations with heterogeneous selectivity. Using high-resolution functional magnetic resonance imaging (MRI), we found that the FFA has a reliable heterogeneous structure: localized subregions within the FFA highly selective to faces are spatially interdigitated with localized subregions highly selective to different object categories. We found a preponderance of face-selective responses in the FFA, but no difference in selectivity to faces compared to nonfaces. Thus, standard fMRI of the FFA reflects averaging of heterogeneous highly selective neural populations of differing sizes, rather than higher selectivity to faces. These results suggest that visual processing in this region is not exclusive to faces. Overall, our approach provides a framework for understanding the fine-scale structure of neural representations in the human brain.     

Human cortical activity correlates with stereoscopic depth perception

Stereoscopic depth perception is based on binocular disparities. Although neurons in primary visual cortex (V1) are selective for binocular disparity, their responses do not explicitly code perceived depth. The stereoscopic pathway must therefore include additional processing beyond V1. We used functional magnetic resonance imaging (fMRI) to examine stereo processing in V1 and other areas of visual cortex. We created stereoscopic stimuli that portrayed two planes of dots in depth, placed symmetrically about the plane of fixation, or else asymmetrically with both planes either nearer or farther than fixation. The interplane disparity was varied parametrically to determine the stereoacuity threshold (the smallest detectable disparity) and the upper depth limit (largest detectable disparity). fMRI was then used to quantify cortical activity across the entire range of detectable interplane disparities. Measured cortical activity covaried with psychophysical measures of stereoscopic depth perception. Activity increased as the interplane disparity increased above the stereoacuity threshold and dropped as interplane disparity approached the upper depth limit. From the fMRI data and an assumption that V1 encodes absolute retinal disparity, we predicted that the mean response of V1 neurons should be a bimodal function of disparity. A post hoc analysis of electrophysiological recordings of single neurons in macaques revealed that, although the average firing rate was a bimodal function of disparity (as predicted), the precise shape of the function cannot fully explain the fMRI data. Although there was widespread activity within the extrastriate cortex (consistent with electrophysiological recordings of single neurons), area V3A showed remarkable sensitivity to stereoscopic stimuli, suggesting that neurons in V3A may play a special role in the stereo pathway.     

Human cortical response to various apparent motions: a magnetoencephalographic study

The human visual system is considered to have at least two different mechanisms for perceiving motions: one for luminance-based (first-order) motions and another for non-luminance-based (second-order) motions. In this study, we examined the perception of first- and second-order motions using four different types of stimulus cues (luminance, contrast, texture, and flicker) while using whole head magnetoencephalography (MEG) to measure human brain responses to those apparent motions. MEG responses to all stimuli were recorded from the occipito-temporal area (possibly human MT/V5+), and response properties (peak latency and amplitude) varied with stimulus cues. Further, we observed various effects of luminance-addition to the non-luminance cues on the response properties that could not be explained by the magnetic field distribution and/or the visibility of the stationary object. The results indicate that differences in response properties elicited by various stimulus cues represent differences in the neural processes underlying apparent motions with various cues. We suggest that the distinct "preprocessing" of each stimulus cue occurs before the common process for apparent motion, and the response property changes associated with different cues are related to differences in preprocessing that may occur in a distributed cortical network that include the striate and extrastriate visual cortex.     

A dynamic fMRI study of illusory double-flash effect on human visual cortex

Functional MRI (fMRI) combined with the paired-stimuli paradigms (referred as dynamic fMRI) was used to study the “illusory double-flash” effect on brain activity in the human visual cortex. Three experiments were designed. The first two experiments aimed to examine the cross-modal neural interaction between the visual and auditory sensory systems caused by the illusory double-flash effect using combined auditory (beep sound) and visual (light flash) stimuli. The fMRI signal in the visual cortex was significantly increased in response to the illusory double flashes compared to the physical single flash when the inter-stimuli delay between the auditory and visual stimuli was 25 ms. This increase disappeared when the delay was prolonged to ~300 ms. These results reveal that the illusory double-flash effect can significantly affect the brain activity in the visual cortex, and the degree of this effect is dynamically sensitive to the inter-stimuli delay. The third experiment was to address the spatial differentiation of brain activation in the visual cortex in response to the illusory double-flash stimulation. It was found that the illusory double-flash effect in the human visual cortex is much stronger in the periphery than the fovea. This finding suggests that the periphery may be involved in high-level brain processing beyond the retinotopic visual perception. The behavioral measures conducted in this study indicate an excellent correlation between the fMRI results and behavioral performance. Finally, this work demonstrates a unique merit of fMRI for providing both temporal and spatial information regarding cross-modal neural interaction between different sensory systems.     

Stereoscopic processing of absolute and relative disparity in human visual cortex

Stereoscopic vision relies mainly on relative depth differences between objects rather than on their absolute distance in depth from where the eyes fixate. However, relative disparities are computed from absolute disparities, and it is not known where these two stages are represented in the human brain. Using functional MRI (fMRI), we assessed absolute and relative disparity selectivity with stereoscopic stimuli consisting of pairs of transparent planes in depth in which the absolute and relative disparity signals could be independently manipulated (at a local spatial scale). In experiment 1, relative disparity was kept constant, while absolute disparity was varied in one-half the blocks of trials ("mixed" blocks) and kept constant in the remaining one-half ("same" blocks), alternating between blocks. Because neuronal responses undergo adaptation and reduce their firing rate following repeated presentation of an effective stimulus, the fMRI signal reflecting activity of units selective for absolute disparity is expected to be smaller during "same" blocks as compared with "mixed" ones. Experiment 2 similarly manipulated relative disparity rather than absolute disparity. The results from both experiments were consistent with adaptation with differential effects across visual areas such that 1) dorsal areas (V3A, MT+/V5, V7) showed more adaptation to absolute than to relative disparity; 2) ventral areas (hV4, V8/V4alpha) showed an equal adaptation to both; and 3) early visual areas (V1, V2, V3) showed a small effect in both experiments. These results indicate that processing in dorsal areas may rely mostly on information about absolute disparities, while ventral areas split neural resources between the two types of stereoscopic information so as to maintain an important representation of relative disparity.     

Using visual direction in three-dimensional motion perception

The eyes receive slightly different views of the world, and the differences between their images (binocular disparity) are used to see depth. Several authors have suggested how the brain could exploit this information for three-dimensional (3D) motion perception, but here we consider a simpler strategy. Visual direction is the angle between the direction of an object and the direction that an observer faces. Here we describe human behavioral experiments in which observers use visual direction, rather than binocular information, to estimate an object's 3D motion even though this causes them to make systematic errors. This suggests that recent models of binocular 3D motion perception may not reflect the strategies that human observers actually use.     

An fMRI study of the functional distinction of neuronal circuits at the sites on ventral visual stream co-activated by visual stimuli of different objects

In functional studies of the human brain using functional magnetic resonance imaging (fMRI) we often observe some functional areas that are commonly activated by different stimulus inputs even when the inputs are of different categories. It is difficult to distinguish by fMRI whether the neuronal circuits activated for processing these inputs are separate and independent ones within the functional area or whether they are mutually interactive and possibly sharing a part of the circuits for processing some common features of the input information. In order to elucidate such property of the neuronal circuits, we used a novel paradigm in which paired input stimuli with varied inter stimulus interval (ISI) were presented during a stimulation period in fMRI experiments. The refractory suppression induced by an input pair depends on ISI as well as the differing characteristics of the input pair. The extent of suppression, an indicator for the interaction between the processing activities of the input pair, can be evaluated by the shape of the time-course of fMRI responses. We examined the functional characteristics of the neuronal circuits in areas that were activated by different inputs of inter-category types, namely face stimuli and building stimuli and also by intra-category input pairs such as different faces. In the ventral visual occipitotemporal region, we found functional areas where neuronal circuits were interacting to process these inputs rather than functioning as separate independent circuits. With this approach, one can probe functional system activity at the neuronal circuit level to learn the characteristics that determine the functional roles of certain brain areas.     

Neural mechanisms of three-dimensional vision

We can see things in three dimensions because the visual system re-constructs the three-dimensional (3D) configurations of objects from their two-dimensional (2D) images projected onto the retinas. The purpose of this paper is to give an overview of the psychological background and recent physiological findings concerning three-dimensional vision. Psychophysical and computational studies have suggested that in the visual system the 3D surface orientation is first estimated independently from individual depth cues--such as binocular disparity, as well as various monocular cues including texture gradients--and then the information from these different depth cues is integrated to construct a generalized representation of the 3D surface geometry. Neurons involved in low-level disparity processing, or the detection of local absolute disparity, were found mainly in the occipital cortex, whereas neurons involved in high-level disparity processing, or the reconstruction of 3D surface orientation through the computation of disparity gradients, were found mainly in the parietal area caudal intraparietal sulcus (CIP). Neurons sensitive to texture gradients, which is one of the major monocular cues, were also found in CIP. The majority of these neurons were sensitive to disparity gradients as well, suggesting their involvement in the computation of 3D surface orientation. In CIP, neurons sensitive to multiple depth cues were widely distributed together with those sensitive to a specific depth cue, suggesting CIP's involvement in the integration of depth information from different sources. In addition, human and monkey imaging studies have indicated convergence of multiple depth cues in CIP. These neurophysiological findings suggest that CIP plays a critical role in 3D vision by constructing a generalized representation of the 3D surface geometry of objects.     

Right parietal cortex is involved in the perception of sound movement in humans

Changes in the delay (phase) and amplitude of sound at the ears are cues for the analysis of sound movement. The detection of these cues depends on the convergence of the inputs to each ear, a process that first occurs in the brainstem. The conscious perception of these cues is likely to involve higher centers. Using novel stimuli that produce different perceptions of movement in the presence of identical phase and amplitude modulation components, we have demonstrated human brain areas that are active specifically during the perception of sound movement. Both functional magnetic resonance imaging (fMRI) and positron emission tomography (PET) demonstrated the involvement of the right parietal cortex in sound movement perception with these stimuli.     

Uncertainty and invariance in the human visual cortex

The way in which input noise perturbs the behavior of a system depends on the internal processing structure of the system. In visual psychophysics, there is a long tradition of using external noise methods (i.e., adding noise to visual stimuli) as tools for system identification. Here, we demonstrate that external noise affects processing of visual scenes at different cortical areas along the human ventral visual pathway, from retinotopic regions to higher occipitotemporal areas implicated in visual shape processing. We found that when the contrast of the stimulus was held constant, the further away from the retinal input a cortical area was the more its activity, as measured with functional magnetic resonance imaging (fMRI), depended on the signal-to-noise ratio (SNR) of the visual stimulus. A similar pattern of results was observed when trials with correct and incorrect responses were analyzed separately. We interpret these findings by extending signal detection theory to fMRI data analysis. This approach reveals the sequential ordering of decision stages in the cortex by exploiting the relation between fMRI response and stimulus SNR. In particular, our findings provide novel evidence that occipitotemporal areas in the ventral visual pathway form a cascade of decision stages with increasing degree of signal uncertainty and feature invariance.     

Neural correlates of evidence accumulation in a perceptual decision task

Sequential sampling models provide a useful framework for understanding human decision making. A key component of these models is an evidence accumulation process in which information is accrued over time to a threshold, at which point a choice is made. Previous neurophysiological studies on perceptual decision making have suggested accumulation occurs only in sensorimotor areas involved in making the action for the choice. Here we investigated the neural correlates of evidence accumulation in the human brain using functional magnetic resonance imaging (fMRI) while manipulating the quality of sensory evidence, the response modality, and the foreknowledge of the response modality. We trained subjects to perform a random dot motion direction discrimination task by either moving their eyes or pressing buttons to make their responses. In addition, they were cued about the response modality either in advance of the stimulus or after a delay. We isolated fMRI responses for perceptual decisions in both independently defined sensorimotor areas and task-defined nonsensorimotor areas. We found neural signatures of evidence accumulation, a higher fMRI response on low coherence trials than high coherence trials, primarily in saccade-related sensorimotor areas (frontal eye field and intraparietal sulcus) and nonsensorimotor areas in anterior insula and inferior frontal sulcus. Critically, such neural signatures did not depend on response modality or foreknowledge. These results help establish human brain areas involved in evidence accumulation and suggest that the neural mechanism for evidence accumulation is not specific to effectors. Instead, the neural system might accumulate evidence for particular stimulus features relevant to a perceptual task.