Stereoscopic processing of absolute and relative disparity in human visual cortex

Stereoscopic vision relies mainly on relative depth differences between objects rather than on their absolute distance in depth from where the eyes fixate. However, relative disparities are computed from absolute disparities, and it is not known where these two stages are represented in the human brain. Using functional MRI (fMRI), we assessed absolute and relative disparity selectivity with stereoscopic stimuli consisting of pairs of transparent planes in depth in which the absolute and relative disparity signals could be independently manipulated (at a local spatial scale). In experiment 1, relative disparity was kept constant, while absolute disparity was varied in one-half the blocks of trials ("mixed" blocks) and kept constant in the remaining one-half ("same" blocks), alternating between blocks. Because neuronal responses undergo adaptation and reduce their firing rate following repeated presentation of an effective stimulus, the fMRI signal reflecting activity of units selective for absolute disparity is expected to be smaller during "same" blocks as compared with "mixed" ones. Experiment 2 similarly manipulated relative disparity rather than absolute disparity. The results from both experiments were consistent with adaptation with differential effects across visual areas such that 1) dorsal areas (V3A, MT+/V5, V7) showed more adaptation to absolute than to relative disparity; 2) ventral areas (hV4, V8/V4alpha) showed an equal adaptation to both; and 3) early visual areas (V1, V2, V3) showed a small effect in both experiments. These results indicate that processing in dorsal areas may rely mostly on information about absolute disparities, while ventral areas split neural resources between the two types of stereoscopic information so as to maintain an important representation of relative disparity.     

Human cortical activity correlates with stereoscopic depth perception

Stereoscopic depth perception is based on binocular disparities. Although neurons in primary visual cortex (V1) are selective for binocular disparity, their responses do not explicitly code perceived depth. The stereoscopic pathway must therefore include additional processing beyond V1. We used functional magnetic resonance imaging (fMRI) to examine stereo processing in V1 and other areas of visual cortex. We created stereoscopic stimuli that portrayed two planes of dots in depth, placed symmetrically about the plane of fixation, or else asymmetrically with both planes either nearer or farther than fixation. The interplane disparity was varied parametrically to determine the stereoacuity threshold (the smallest detectable disparity) and the upper depth limit (largest detectable disparity). fMRI was then used to quantify cortical activity across the entire range of detectable interplane disparities. Measured cortical activity covaried with psychophysical measures of stereoscopic depth perception. Activity increased as the interplane disparity increased above the stereoacuity threshold and dropped as interplane disparity approached the upper depth limit. From the fMRI data and an assumption that V1 encodes absolute retinal disparity, we predicted that the mean response of V1 neurons should be a bimodal function of disparity. A post hoc analysis of electrophysiological recordings of single neurons in macaques revealed that, although the average firing rate was a bimodal function of disparity (as predicted), the precise shape of the function cannot fully explain the fMRI data. Although there was widespread activity within the extrastriate cortex (consistent with electrophysiological recordings of single neurons), area V3A showed remarkable sensitivity to stereoscopic stimuli, suggesting that neurons in V3A may play a special role in the stereo pathway.     

Neural networks active during tactile form perception: common and differential activity during macrospatial and microspatial tasks.

Prior studies have shown that tactile perception recruits activity not only in somatosensory but also in visual cortical areas. The present study used functional magnetic resonance imaging to investigate the distribution of neural activity during tactile perception of 2D form. In a macrospatial form task, raised letters (uppercase T and V) were presented upside-down. In a microspatial form task, a bar, either with or without a gap, was presented. Stimuli were applied to the immobilized right index fingerpad. Six neurologically normal volunteers were studied in a block design paradigm, with alternating blocks of rest and covert discrimination between the two alternatives for a task. Each task was studied in a separate run. Contrasting macrospatial form discrimination against rest revealed activity in an extensive, bilateral network of cortical and subcortical regions, including areas of somatosensory cortex and the intraparietal sulcus (IPS), occipito-temporal cortex, dorsal and ventral premotor cortex, medial superior frontal cortex, lateral inferior frontal cortex, thalamus and cerebellar hemispheres. Contrasting (microspatial) gap detection against rest showed activity in a similar network, with the notable exception of the occipito-temporal cortical regions. A direct contrast between the two tasks yielded greater activity for the macrospatial than microspatial task in these occipito-temporal regions bilaterally, and also in foci near the right IPS and in the right cerebellar hemisphere. The occipito-temporal cortical activations were in the lateral occipital complex, a part of the ventral visual pathway active during visual form perception. Thus, macrospatial form perception preferentially recruits this region of extrastriate visual cortex, compared to microspatial form perception.     

Synchronization between the end stages of the dorsal and the ventral visual stream

The end stage areas of the ventral (IT) and the dorsal (AIP) visual streams encode the shape of disparity-defined three-dimensional (3D) surfaces. Recent anatomical tracer studies have found direct reciprocal connections between the 3D-shape selective areas in IT and AIP. Whether these anatomical connections are used to facilitate 3D-shape perception is still unknown. We simultaneously recorded multi-unit activity (MUA) and local field potentials in IT and AIP while monkeys discriminated between concave and convex 3D shapes and measured the degree to which the activity in IT and AIP synchronized during the task. We observed strong beta-band synchronization between IT and AIP preceding stimulus onset that decreased shortly after stimulus onset and became modulated by stereo-signal strength and stimulus contrast during the later portion of the stimulus period. The beta-coherence modulation was unrelated to task-difficulty, regionally specific, and dependent on the MUA selectivity of the pairs of sites under study. The beta-spike-field coherence in AIP predicted the upcoming choice of the monkey. Several convergent lines of evidence suggested AIP as the primary source of the AIP-IT synchronized activity. The synchronized beta activity seemed to occur during perceptual anticipation and when the system has stabilized to a particular perceptual state but not during active visual processing. Our findings demonstrate for the first time that synchronized activity exists between the end stages of the dorsal and ventral stream during 3D-shape discrimination.     

A stereoscopic look at visual cortex

Three recent studies offer new insights into the way visual cortex handles binocular disparity signals. Two of these studies recorded from single neurons in two different visual areas of the monkey brain, one (V5/MT) in dorsal and one (V4) in ventral cortex. While V5/MT neurons respond similarly to neurons in primary visual cortex (V1), V4 neurons appear to reflect a more advanced stage in the analysis of retinal disparity, closer to the perceptual experience of stereoscopic depth. Both studies are consistent with a third study using fMRI to address similar questions in humans. Together with previous evidence, these results suggest a new framework for understanding stereoscopic processing based on the separation between ventral and dorsal streams in visual cortex.     

3D shape perception from combined depth cues in human visual cortex

Our perception of the world's three-dimensional (3D) structure is critical for object recognition, navigation and planning actions. To accomplish this, the brain combines different types of visual information about depth structure, but at present, the neural architecture mediating this combination remains largely unknown. Here, we report neuroimaging correlates of human 3D shape perception from the combination of two depth cues. We measured fMRI responses while observers judged the 3D structure of two sequentially presented images of slanted planes defined by binocular disparity and perspective. We compared the behavioral and fMRI responses evoked by changes in one or both of the depth cues. fMRI responses in extrastriate areas (hMT+/V5 and lateral occipital complex), rather than responses in early retinotopic areas, reflected differences in perceived 3D shape, suggesting 'combined-cue' representations in higher visual areas. These findings provide insight into the neural circuits engaged when the human brain combines different information sources for unified 3D visual perception.     

The role of temporo-parietal junction (TPJ) in global Gestalt perception

Grouping processes enable the coherent perception of our environment. A number of brain areas has been suggested to be involved in the integration of elements into objects including early and higher visual areas along the ventral visual pathway as well as motion-processing areas of the dorsal visual pathway. However, integration not only is required for the cortical representation of individual objects, but is also essential for the perception of more complex visual scenes consisting of several different objects and/or shapes. The present fMRI experiments aimed to address such integration processes. We investigated the neural correlates underlying the global Gestalt perception of hierarchically organized stimuli that allowed parametrical degrading of the object at the global level. The comparison of intact versus disturbed perception of the global Gestalt revealed a network of cortical areas including the temporo-parietal junction (TPJ), anterior cingulate cortex and the precuneus. The TPJ location corresponds well with the areas known to be typically lesioned in stroke patients with simultanagnosia following bilateral brain damage. These patients typically show a deficit in identifying the global Gestalt of a visual scene. Further, we found the closest relation between behavioral performance and fMRI activation for the TPJ. Our data thus argue for a significant role of the TPJ in human global Gestalt perception.     

Task-related modulation of visual cortex

We performed a series of experiments to quantify the effects of task performance on cortical activity in early visual areas. Functional magnetic resonance imaging (fMRI) was used to measure cortical activity in several cortical visual areas including primary visual cortex (V1) and the MT complex (MT+) as subjects performed a variety of threshold-level visual psychophysical tasks. Performing speed, direction, and contrast discrimination tasks produced strong modulations of cortical activity. For example, one experiment tested for selective modulations of MT+ activity as subjects alternated between performing contrast and speed discrimination tasks. MT+ responses modulated in phase with the periods of time during which subjects performed the speed discrimination task; that is, MT+ activity was higher during speed discrimination than during contrast discrimination. Task-related modulations were consistent across repeated measurements in each subject; however, significant individual differences were observed between subjects. Together, the results suggest 1) that specific changes in the cognitive/behavioral state of a subject can exert selective and reliable modulations of cortical activity in early visual cortex, even in V1; 2) that there are significant individual differences in these modulations; and 3) that visual areas and pathways that are highly sensitive to small changes in a given stimulus feature (such as contrast or speed) are selectively modulated during discrimination judgments on that feature. Increasing the gain of the relevant neuronal signals in this way may improve their signal-to-noise to help optimize task performance.     

Emotional perception: Correspondence of early and late event-related potentials with cortical and subcortical functional MRI

This research examines the relationship between brain activity recorded with functional magnetic resonance imaging (fMRI) and event related potentials (ERP) as these responses varied over a series of emotionally evocative and neutral pictures. We investigate the relationship of early occipitotemporal and later centroparietal emotion-modulated ERPs in one sample to fMRI estimates of neural activity in another sample in a replicated experiment. Using this approach, we aimed to link effects found in time-resolved electrocortical measures to specific neural structures across individual emotional and nonemotional picture stimuli. The centroparietal late positive potential (LPP) showed covariation with emotion-modulated regions of hemodynamic activation across multiple dorsal and ventral visual cortical structures, while the early occipitotemporal potential was not reliably associated. Subcortical and corticolimbic structures involved in the perception of motivationally relevant stimuli also related to modulation of the LPP, and were modestly associated to the amplitude of the early occipitotemporal potential. These data suggest that early occipitotemporal potentials may reflect multiple sources of modulation including motivational relevance, and supports the perspective that the slow-wave LPP represents aggregate cortical and subcortical structures involved in emotional discrimination.     

Projections from the cytochrome oxidase modules of visual area V2 to the ventral posterior area in the macaque

The ventral part of the third visual cortical complex, the ventral posterior area (VP) or V3v, is located between the ventral half of visual areas V2 and V4. Because of its location and the physiological properties of its neurons, VP has been considered to be involved in the ventral stream visual areas. The ventral stream visual areas such as V4 and TEO receive projections from the cytochrome oxidase (CO)-rich thin stripes and CO-poor interstripe regions of V2; however, which CO-modules project to VP remains unclear. Moreover, it is not clear whether V1 projects to VP. We injected retrograde tracers into VP and found that VP receives projections from V2 neurons not only in the CO-rich thin stripes and CO-poor interstripe regions but also in the CO-rich thick stripes. We also confirmed the virtual absence of inputs from V1 to VP. These results support the hypothesis that VP constitutes a distinct extrastriate visual area and also suggest that, in addition to color and shape information, VP may also process visual information related to space and disparity.     

Cortical responses to invisible objects in the human dorsal and ventral pathways

The primate visual system is believed to comprise two main pathways: a ventral pathway for conscious perception and a dorsal pathway that can process visual information and guide action without accompanying conscious knowledge. Evidence for this theory has come primarily from studies of neurological patients and animals. Using fMRI, we show here that even though observers are completely unaware of test object images owing to interocular suppression, their dorsal cortical areas demonstrate substantial activity for different types of visual objects, with stronger responses to images of tools than of human faces. This result also suggests that in binocular rivalry, substantial information in the suppressed eye can escape the interocular suppression and reach dorsal cortex.     

Short-term visual deprivation alters neural processing of tactile form

Blindness is known to alter the responsiveness of visual cortex. Recently, reversible visual deprivation by blindfolding has been shown to affect non-visual abilities as well as visual cortical function. Here we investigated the effect of 2 h of blindfolding on cerebral cortical activation patterns during tactile form perception, using functional magnetic resonance imaging. Two form tasks were used, one requiring discrimination of global stimulus form and the other, detection of a gap in a bar. Blindfolded subjects showed significant deactivation during these tasks in regions that are intermediate in the hierarchy of visual shape processing: probable V3A and ventral intraparietal sulcus (vIPS). These regions lacked signal changes in controls. There were also task-specific increases in activation in blindfolded relative to control subjects, favoring the form over the gap task, along the IPS and in regions of frontal and temporal cortex. We also found alterations of functional connectivity that corresponded to the activity differences, with the emergence of correlated activity between the vIPS and V3A in blindfolded subjects. We conclude that blindfolding sighted individuals for a 2-h period induces significant changes in the neural processing of tactile form, probably reflecting short-term neural plasticity.     

Uncertainty and invariance in the human visual cortex

The way in which input noise perturbs the behavior of a system depends on the internal processing structure of the system. In visual psychophysics, there is a long tradition of using external noise methods (i.e., adding noise to visual stimuli) as tools for system identification. Here, we demonstrate that external noise affects processing of visual scenes at different cortical areas along the human ventral visual pathway, from retinotopic regions to higher occipitotemporal areas implicated in visual shape processing. We found that when the contrast of the stimulus was held constant, the further away from the retinal input a cortical area was the more its activity, as measured with functional magnetic resonance imaging (fMRI), depended on the signal-to-noise ratio (SNR) of the visual stimulus. A similar pattern of results was observed when trials with correct and incorrect responses were analyzed separately. We interpret these findings by extending signal detection theory to fMRI data analysis. This approach reveals the sequential ordering of decision stages in the cortex by exploiting the relation between fMRI response and stimulus SNR. In particular, our findings provide novel evidence that occipitotemporal areas in the ventral visual pathway form a cascade of decision stages with increasing degree of signal uncertainty and feature invariance.     

The integration of motion and disparity cues to depth in dorsal visual cortex

Humans exploit a range of visual depth cues to estimate three-dimensional structure. For example, the slant of a nearby tabletop can be judged by combining information from binocular disparity, texture and perspective. Behavioral tests show humans combine cues near-optimally, a feat that could depend on discriminating the outputs from cue-specific mechanisms or on fusing signals into a common representation. Although fusion is computationally attractive, it poses a substantial challenge, requiring the integration of quantitatively different signals. We used functional magnetic resonance imaging (fMRI) to provide evidence that dorsal visual area V3B/KO meets this challenge. Specifically, we found that fMRI responses are more discriminable when two cues (binocular disparity and relative motion) concurrently signal depth, and that information provided by one cue is diagnostic of depth indicated by the other. This suggests a cortical node important when perceiving depth, and highlights computations based on fusion in the dorsal stream.     

Disparity selectivity of neurons in monkey inferior temporal cortex

The inferior temporal cortex (IT) of the monkey, a final stage in the ventral visual pathway, has been known to process information on two-dimensional (2-D) shape, color, and texture. On the other hand, the dorsal visual pathway leading to the posterior parietal cortex has been known to process information on location in space. Likewise, neurons selective for binocular disparity, which convey information on depth, have been found mainly in areas along the dorsal visual pathway. Here, we report that many neurons in the IT are also selective for binocular disparity. We recorded extracellular activity from IT neurons and found that more than half of the neurons changed their response depending on the disparity added. The change was not attributed to monocular responses or eye movements. Most neurons selective for disparity were "near" or "far" cells; they preferred either crossed or uncrossed disparity, and only a small population was tuned to zero disparity. Disparity-selective neurons were also selective for shape. Most preferred the same type of disparity irrespective of the shape presented. Disparity preference was also invariant for the fronto-parallel translation of the stimuli in most of the neurons. Finally, nearby neurons exhibited similar disparity selectivity, suggesting the existence of a functional module for processing of binocular disparity in the IT. From the above and our recent findings, we suggest that the IT integrates shape and binocular disparity information, and plays an important role in the reconstruction of three-dimensional (3-D) surfaces.     

Visually guided grasping produces fMRI activation in dorsal but not ventral stream brain areas

Although both reaching and grasping require transporting the hand to the object location, only grasping also requires processing of object shape, size and orientation to preshape the hand. Behavioural and neuropsychological evidence suggests that the object processing required for grasping relies on different neural substrates from those mediating object recognition. Specifically, whereas object recognition is believed to rely on structures in the ventral (occipitotemporal) stream, object grasping appears to rely on structures in the dorsal (occipitoparietal) stream. We used functional magnetic resonance imaging (fMRI) to determine whether grasping (compared to reaching) produced activation in dorsal areas, ventral areas, or both. We found greater activity for grasping than reaching in several regions, including anterior intraparietal (AIP) cortex. We also performed a standard object perception localizer (comparing intact vs. scrambled 2D object images) in the same subjects to identify the lateral occipital complex (LOC), a ventral stream area believed to play a critical role in object recognition. Although LOC was activated by the objects presented on both grasping and reaching trials, there was no greater activity for grasping compared to reaching. These results suggest that dorsal areas, including AIP, but not ventral areas such as LOC, play a fundamental role in computing object properties during grasping.     

Mapping brain activation and information during category-specific visual working memory

How is working memory for different visual categories supported in the brain? Do the same principles of cortical specialization that govern the initial processing and encoding of visual stimuli also apply to their short-term maintenance? We investigated these questions with a delayed discrimination paradigm for faces, bodies, flowers, and scenes and applied both univariate and multivariate analyses to functional magnetic resonance imaging (fMRI) data. Activity during encoding followed the well-known specialization in posterior areas. During the delay interval, activity shifted to frontal and parietal regions but was not specialized for category. Conversely, activity in visual areas returned to baseline during that interval but showed some evidence of category specialization on multivariate pattern analysis (MVPA). We conclude that principles of cortical activation differ between encoding and maintenance of visual material. Whereas perceptual processes rely on specialized regions in occipitotemporal cortex, maintenance involves the activation of a frontoparietal network that seems to require little specialization at the category level. We also confirm previous findings that MVPA can extract information from fMRI signals in the absence of suprathreshold activation and that such signals from visual areas can reflect the material stored in memory.     

Two hierarchically organized neural systems for object information in human visual cortex

The primate visual system is broadly organized into two segregated processing pathways, a ventral stream for object vision and a dorsal stream for space vision. Here, evidence from functional brain imaging in humans demonstrates that object representations are not confined to the ventral pathway, but can also be found in several areas along the dorsal pathway. In both streams, areas at intermediate processing stages in extrastriate cortex (V4, V3A, MT and V7) showed object-selective but viewpoint- and size-specific responses. In contrast, higher-order areas in lateral occipital and posterior parietal cortex (LOC, IPS1 and IPS2) responded selectively to objects independent of image transformations. Contrary to the two-pathways hypothesis, our findings indicate that basic object information related to shape, size and viewpoint may be represented similarly in two parallel and hierarchically organized neural systems in the ventral and dorsal visual pathways.     

Filling-in of visual phantoms in the human brain

The constructive nature of perception can be demonstrated under viewing conditions that lead to vivid subjective impressions in the absence of direct input. When a low-contrast moving grating is divided by a large gap, observers report seeing a 'visual phantom' of the real grating extending through the blank gap region. Here, we report fMRI evidence showing that visual phantoms lead to enhanced activity in early visual areas that specifically represent the blank gap region. We found that neural filling-in effects occurred automatically in areas V1 and V2, regardless of where the subject attended. Moreover, when phantom-inducing gratings were paired with competing stimuli in a binocular rivalry display, subjects reported spontaneous fluctuations in conscious perception of the phantom accompanied by tightly coupled changes in early visual activity. Our results indicate that phantom visual experiences are closely linked to automatic filling-in of activity at the earliest stages of cortical processing.     

Dissociating cortical regions activated by semantic and phonological tasks: a FMRI study in blind and sighted people

Previous neuroimaging studies of language processing in blind individuals described cortical activation of primary (V1) and higher tier visual areas, irrespective of the age of blindness onset. Specifically, participants were given nouns and asked to generate an associated verb. These results confirmed the presence of adaptations in the visual cortex of blind people and suggested that these responses represented linguistic operations. The present functional magnetic resonance imaging study attempted to further characterize these responses as being preferential for semantic or phonological processing. Three groups of participants (sighted, early onset, and late-onset blind) heard lists of related words and attended to either a common meaning (semantic task) or common rhyme (phonological task) that linked the words. In all three groups, the semantic task elicited stronger activity in the left anterior inferior frontal gyrus and the phonological task evoked stronger activity bilaterally in the inferior parietal cortex and posterior aspects of the left inferior frontal gyrus. Only blind individuals showed activity in occipital, temporal, and parietal components of visual cortex. The spatial extent of visual cortex activity was greatest in early blind, who exhibited activation in all ventral and dorsal visual cortex subdivisions (V1 through MT) for both tasks. Preferential activation appeared for the semantic task. Late blind individuals exhibited responses in ventral and dorsal V1, ventral V2, VP and V8, but only for the semantic task. Our findings support prior evidence of visual cortex activity in blind people engaged in auditory language processing and suggest that this activity may be related to semantic processing.