Cross-modal binding and activated attentional networks during audio-visual speech integration: a functional MRI study

We evaluated the neural substrates of cross-modal binding and divided attention during audio-visual speech integration using functional magnetic resonance imaging. The subjects (n = 17) were exposed to phonemically concordant or discordant auditory and visual speech stimuli. Three different matching tasks were performed: auditory-auditory (AA), visual-visual (VV) and auditory-visual (AV). Subjects were asked whether the prompted pair were congruent or not. We defined the neural substrates for the within-modal matching tasks by VV-AA and AA-VV. We defined the cross-modal area as the intersection of the loci defined by AV-AA and AV-VV. The auditory task activated the bilateral anterior superior temporal gyrus and superior temporal sulcus, the left planum temporale and left lingual gyrus. The visual task activated the bilateral middle and inferior frontal gyrus, right occipito-temporal junction, intraparietal sulcus and left cerebellum. The bilateral dorsal premotor cortex, posterior parietal cortex (including the bilateral superior parietal lobule and the left intraparietal sulcus) and right cerebellum showed more prominent activation during AV compared with AA and VV. Within these areas, the posterior parietal cortex showed more activation during concordant than discordant stimuli, and hence was related to cross-modal binding. Our results indicate a close relationship between cross-modal attentional control and cross-modal binding during speech reading.     

Cross-modal integration and plastic changes revealed by lip movement, random-dot motion and sign languages in the hearing and deaf

Sign language activates the auditory cortex of deaf subjects, which is evidence of cross-modal plasticity. Lip-reading (visual phonetics), which involves audio-visual integration, activates the auditory cortex of hearing subjects. To test whether audio-visual cross-modal plasticity occurs within areas involved in cross-modal integration, we used functional MRI to study seven prelingual deaf signers, 10 hearing non-signers and nine hearing signers. The visually presented tasks included mouth-movement matching, random-dot motion matching and sign-related motion matching. The mouth-movement tasks included conditions with or without visual phonetics, and the difference between these was used to measure the lip-reading effects. During the mouth-movement matching tasks, the deaf subjects showed more prominent activation of the left planum temporale (PT) than the hearing subjects. During dot-motion matching, the deaf showed greater activation in the right PT. Sign-related motion, with or without a lexical component, activated the left PT in the deaf signers more than in the hearing signers. These areas showed lip-reading effects in hearing subjects. These findings suggest that cross-modal plasticity is induced by auditory deprivation independent of the lexical processes or visual phonetics, and this plasticity is mediated in part by the neural substrates of audio-visual cross-modal integration.     

Distribution of cortical activation during visuospatial n-back tasks as revealed by functional magnetic resonance imaging

Human neuroimaging studies conducted during visuospatial working memory tasks have inconsistently detected activation in the prefrontal cortical areas depending presumably on the type of memory and control tasks employed. We used functional magnetic resonance imaging to study brain activation related to the performance of a visuospatial n-back task with different memory loads (0-back, 1-back and 2-back tasks). Comparison of the 2-back versus 0-back tasks revealed consistent, bilateral activation in the medial frontal gyrus (MFG), superior frontal sulcus and adjacent cortical tissue (SFS/SFG) in all subjects and in six out of seven subjects in the intraparietal sulcus (IPS). Activation was also detected in the inferior frontal gyrus, medially in the superior frontal gyrus, precentral gyrus, superior and inferior parietal lobuli, occipital visual association areas, anterior and posterior cingulate areas and in the insula. Comparison between the 1- back versus 0-back tasks revealed activation only in a few brain areas. Activation in the MFG, SFS/SFG and IPS appeared dependent on memory load. The results suggest that the performance of a visuospatial working memory task engages a network of distributed brain areas and that areas in the dorsal visual pathway are engaged in mnemonic processing of visuospatial information.      

Distributed and associative working memory

This study explores the cortical cell dynamics of unimodal and cross-modal working memory (WM). Neuronal activity was recorded from parietal areas of monkeys performing delayed match-to-sample tasks with tactile or visual samples. Tactile memoranda (haptic samples) consisted of rods with differing surface features (texture or orientation of ridges) perceived by active touch. Visual memoranda (icons) consisted of striped patterns of differing orientation. In a haptic-haptic task, the animal had to retain through a period of delay the surface feature of the sample rod to select a rod that matched it. In a visual-haptic task, the animal had to retain the icon for the haptic choice of a rod with ridges of the same orientation as the icon's stripes. Units in all areas responded with firing change to one or more task events. Also in all areas, cells responded differently to different sample memoranda. Differential sample coherent firing was present in most areas during the memory period (delay). It is concluded that neurons in somatosensory and association areas of parietal cortex participate in broad networks that represent various task events and stimuli (auditory, motor, proprioceptive, tactile, and visual). Neurons in the same networks take part in retaining in WM the memorandum for each trial, whether it is encoded haptically or visually. The VH association by parietal cells in WM is analogous to the auditory-visual association previously observed in prefrontal cortex. Both illustrate the capacity of cortical neurons to associate sensory information across time and across modalities in accord with the rules of a behavioral task.     

Concurrent performance of two working memory tasks: potential mechanisms of interference

When two tasks are performed simultaneously, performance often deteriorates, with concomitant increases in reaction time and error rate. Three potential neurophysiological mechanisms behind this deterioration in performance have been considered here: (i) dual-task performance requires additional cognitive operations and activation of cortical areas in addition to those active during single-task performance; (ii) two tasks interfere if they require activation of the same part of cortex; and (iii) cross-modal inhibition causes interference between two tasks involving stimuli from different sensory modalities. Positron emission tomography was used to measure regional cerebral blood flow (rCBF) during performance of an auditory working memory (WM) task, a visual WM task, both WM tasks (dual task) and a control condition. Compared to the control condition, the auditory and visual WM tasks activated sensory-specific areas in the superior temporal gyrus and occipital pole respectively. Both WM tasks also activated overlapping parts of cortex in the dorsolateral prefrontal, inferior parietal and cingulate cortex. There was no separate cortical area which was activated only in the dual task, and thus no area which could be associated with any dual task specific cognitive process such as task-coordination or divided attention. Decrease in rCBF in one WM task did not overlap with the areas of rCBF increase in the other WM task. However, an inhibitory mechanism could not be ruled out, since the rCBF increase in sensory specific areas was smaller in the dual- task condition than in the single-task conditions. The cortical activity underlying WM was to a large extent organized in a non-sensory specific, or non-parallel, way, and the results are consistent with the hypothesis that concurrent tasks interfere with each other if they demand activation of the same part of cortex.      

Functional cerebral reorganization for auditory spatial processing and auditory substitution of vision in early blind subjects

Early blind (EB) individuals can recognize bidimensional shapes using a prosthesis substituting vision with audition (PSVA) and activate right dorsal extrastriate visual cortex during the execution of this task. The present study used repetitive transcranial magnetic stimulation (rTMS) to further examine the functional role of this structure in the successful use of the PSVA. Moreover, we investigated which auditory parameter used in the prosthesis (pitch, intensity, or spatial location) might contribute to this occipital activation. Results revealed that rTMS applied to right dorsal extrastriate cortex in EB subjects interferes with both the PSVA use and the auditory spatial location task but not with pitch and intensity discriminations. By contrast, rTMS targeting the same cortical areas in sighted subjects did not affect performance on any auditory tasks. Early visual deprivation thus leads to functional cerebral cross-modal reorganization in the processing of auditory information and auditory-to-visual sensory substitution. The findings also point to the specific involvement of the dorsal visual stream for auditory spatial processing in blind subjects. Moreover, this suggests that sensory substitution prostheses can be developed using these additional neural resources to perform tasks that partially compensate for the loss of vision.     

Bilateral activation of fronto-parietal networks by incrementing demand in a working memory task

Working memory (WM) is known to activate the prefrontal cortex. In the present study we hypothesized that when additional contingencies are added to the instruction of a WM task, this would increase the WM load and result in the activation of additional prefrontal areas. With positron emission tomography we measured regional cerebral blood flow in nine subjects performing a control task and two delayed matching to sample tasks, in which the subjects were matching colours and patterns to a reference picture. The second of the two delayed matching tasks had a more complex instruction than the first, with additional contingencies of how to alternate between the matching of colours and patterns. This task thus required the subjects not only to remember a stimulus to match but also to perform this matching according to a specified plan. Both delayed matching tasks activated cortical fields in the middle frontal gyrus, the frontal operculum, upper cingulate gyrus, inferior parietal cortex and cortex lining the intraparietal sulcus, all in the left hemisphere. When alternated delayed matching was compared to simple delayed matching, increases were located in the right superior and middle frontal gyrus and the right anterior inferior parietal cortex. The increased demand during alternated matching thus resulted in bilateral activation of both dorsolateral prefrontal and inferior parietal cortex. The area in the inferior parietal cortex has previously been coactivated with the dorsolateral prefrontal cortex in several WM tasks, irrespective of the sensory modality of the stimuli, and during tasks involving planning.      

Dissociable functional cortical topographies for working memory maintenance of voice identity and location

In order to ascertain whether the neural system for auditory working memory exhibits a functional dissociation for spatial and nonspatial information, we used functional magnetic resonance imaging and a single set of auditory stimuli to study working memory for the location and identity of human voices. The subjects performed a delayed recognition task for human voices and voice locations and an auditory sensorimotor control task. Several temporal, parietal, and frontal areas were activated by both memory tasks in comparison with the control task. However, during the delay periods, activation was greater for the location than for the voice identity task in dorsal prefrontal (SFS/PreCG) and parietal regions and, conversely, greater for voices than locations in ventral prefrontal cortex and the anterior portion of the insula. This preferential response to the voice identity task in ventral prefrontal cortex continued during the recognition test period, but the double dissociation was observed only during maintenance, not during encoding or recognition. Together, the present findings suggest that, during auditory working memory, maintenance of spatial and nonspatial information modulates activity preferentially in a dorsal and a ventral auditory pathway, respectively. Furthermore, the magnitude of this dissociation seems to be dependent on the cognitive operations required at different times during task performance.     

Separate neural correlates for the mnemonic components of successive discrimination and working memory tasks

We have used positron emission tomography to map the mnemonic components of two tasks at the extremes of the visual short-term/ working memory spectrum. The successive discrimination task requires only storage of a single item for very short time (ultra-short- term memory), while the 2back task requires both maintenance (i.e. storage and rehearsal) and manipulation of several items (working memory). We tested whether or not the storage component, common to the two tasks, engaged the same cerebral regions. To remove unnecessary confounds, we reduced the cues available to the subjects to a single elementary attribute, the orientation of a grating presented in central vision. This prevented subjects from using verbal strategies or vestibular cues and allowed equating of difficulty among tasks. Ultra-short-term memory for orientation engaged a large expanse of occipito-temporal cortex with a rate-dependent antero-posterior gradient: a fast trial rate engaged posterior regions, a slow trial rate anterior regions. On the other hand, working memory for orientation involved the left inferior parietal cortex, left dorsolateral prefrontal cortex and a left superior frontal sulcus region, and to a lesser degree the symmetrical right superior frontal region and a left superior parietal region. Direct comparison of the two orientation memory networks confirmed their functional segregation. We conclude that at least the storage of orientation information engages distinct regions depending on whether or not short-term memory/working memory involves rehearsal and/or manipulative processes.     

Distinct causal influences of parietal versus frontal areas on human visual cortex: evidence from concurrent TMS-fMRI

It has often been proposed that regions of the human parietal and/or frontal lobe may modulate activity in visual cortex, for example, during selective attention or saccade preparation. However, direct evidence for such causal claims is largely missing in human studies, and it remains unclear to what degree the putative roles of parietal and frontal regions in modulating visual cortex may differ. Here we used transcranial magnetic stimulation (TMS) and functional magnetic resonance imaging (fMRI) concurrently, to show that stimulating right human intraparietal sulcus (IPS, at a site previously implicated in attention) elicits a pattern of activity changes in visual cortex that strongly depends on current visual context. Increased intensity of IPS TMS affected the blood oxygen level-dependent (BOLD) signal in V5/MT+ only when moving stimuli were present to drive this visual region, whereas TMS-elicited BOLD signal changes were observed in areas V1-V4 only during the absence of visual input. These influences of IPS TMS upon remote visual cortex differed significantly from corresponding effects of frontal (eye field) TMS, in terms of how they related to current visual input and their spatial topography for retinotopic areas V1-V4. Our results show directly that parietal and frontal regions can indeed have distinct patterns of causal influence upon functional activity in human visual cortex.     

A discrete area within the left dorsolateral prefrontal cortex involved in visual-verbal incongruence judgment

The role of the frontal lobe in cross-modal visual-auditory processing has been documented in experiments using incongruent/congruent paradigms. In this study, 4 patients with left frontal World Health Organization Grade II glioma were assessed during pre-, intra-, and postoperative sessions with picture-naming and verbal-visual task requiring judgment of congruence between pictures and words. During awake brain surgery, the naming and cross-modal tasks were coupled with electrical stimulation inactivating restricted specific regions. For all patients, focal brain stimulation in the dorsolateral prefrontal cortex elicited picture-word matching disturbances but no naming impairment, and the elicited errors exclusively appeared in incongruent and not congruent conditions. The dissociation observed between correct picture naming and disturbed cross-modal judgment shows that electrical stimulation of a discrete cortical area within the left dorsolateral prefrontal cortex can inhibit the simultaneous processing of visual-verbal information without disturbing larger networks involved in the naming process.     

Active Representation of Shape and Spatial Location in Man

Neural activity during the delay period of spatial delayed response(DR) and delayed matching (DM) tasks was investigated by positronemission tomography. A distributed cortical system was activatedin each condition. The bilateral dorsolateral prefrontal cortex(DLPFC) was activated in the delay period of both tasks; activationwas of higher significance on the right in the DR task and theleft in the DM task, and extended to the anterolateral prefrontalcortex in the DM condition. Active representation of spatiallocation in the DR task was associated with co-activation ofthe medial and lateral parietal cortex and the extrastriatevisual cortex. Active representation of shape in the DM taskwas associated with co-activation of the medial and lateralparietal cortex and the inferior temporal cortex. Response-relatedactivity was observed in both tasks. Activation of anteriorcingulate, inferior frontal, lateral premotor and rostral inferiorparietal cortex was observed in the DR condition, a task characterizedby preparation of a movement to a predetermined location. Incontrast, preparation to move to an undetermined location inthe DM task was associated with activation predominantly inrostral SMA.     

Working memory specific activity in auditory cortex: potential correlates of sequential processing and maintenance

Working memory (WM) tasks involve several interrelated processes during which past information must be transiently maintained, recalled, and compared with test items according to previously instructed rules. It is not clear whether the rule-specific comparisons of perceptual with memorized items are only performed in previously identified frontal and parietal WM areas or whether these areas orchestrate such comparisons by feedback to sensory cortex. We tested the latter hypothesis by focusing on auditory cortex (AC) areas with low-noise functional magnetic resonance imaging in a 2-back WM task involving frequency-modulated (FM) tones. The control condition was a 0-back task on the same stimuli. Analysis of the group data identified an area on right planum temporale equally activated by both tasks and an area on the left planum temporale specifically involved in the 2-back task. A region of interest analysis in each individual revealed that activation on the left planum temporale in the 2-back task positively correlated with the task performance of the subjects. This strongly suggests a prominent role of the AC in 2-back WM tasks. In conjunction with previous findings on FM processing, the left lateralized effect presumably reflects the complex sequential processing demand of the 2-back matching to sample task.     

Inferior parietal lobule supports decision making under uncertainty in humans

The optimal responses for many decisions faced by humans are ill defined, yet we are able to choose well by associating choices with outcomes, and employing this information in decision making. Previous studies suggest that the parietal cortex is involved in "uncertain" decision making, yet uncertainty is confounded with increased difficulty and attention. Here we aim to dissociate the role of parietal cortex in decision making and attention. Using functional magnetic resonance imaging we measured brain activity while participants played a "matching-pennies" game. We found that the inferior parietal lobule is involved in decision making under uncertainty, showing higher activity when the decision was uncertain rather than certain and when humans were given trial-by-trial feedback on choice outcomes than when they were not. Crucially, increasing attentional load with secondary tasks reduced inferior parietal activity when decisions were made under uncertainty, suggesting that general attention does not drive its activation. This pattern was consistent for visual or auditory feedback, and for direct (symbols representing wins and losses) or indirect (only the opponent's choices were shown) feedback. It contrasted with results from medial superior frontal gyrus, which was driven primarily by increased attentional load. We suggest that decision making under uncertainty is dissociable from general attention in the brain.     

Two attentional processes in the parietal lobe

We report fMRI evidence for two attentional processes in parietal cortex. Subjects matched a feature, cued by a word, to a test display of moving colored dots. Either color (red, green) or motion direction (left, right) was cued on mixed scans while only one dimension was cued on blocked scans. An event-related paradigm separated the preparatory activity generated by the cue from the subsequent activity related to the test display. One attentional process specified task information while a second process was motion selective. During the cue period, a pure effect of task specification was observed in left frontal cortex while combined effects of task specification and motion selectivity were observed in left posterior parietal cortex. The frontal task-specification signal may have been the source of the corresponding signal in parietal cortex. Effects of task specification generalized over cue dimension, indicating that the information was coded in a sufficiently abstract form to affect color and motion processing. During the subsequent test period, task-specification and motion-selective signals were again observed in left parietal cortex. Task specification did not significantly affect occipital motion-selective regions, such as MT+, however, indicating that this process did not influence the lower cortical tier of the motion processing stream. These results provide evidence for general and specialized task representations within left parietal cortex during task preparation and execution.     

Hearing lips: gamma-band activity during audiovisual speech perception

Auditory pattern changes have been shown to elicit increases in magnetoencephalographic gamma-band activity (GBA) over left inferior frontal cortex, forming part of the putative auditory ventral "what" processing stream. The present study employed a McGurk-type paradigm to assess whether GBA would be associated with subjectively perceived changes even when auditory stimuli remain unchanged. Magnetoencephalograms were recorded in 16 human subjects during audiovisual mismatch perception. Both infrequent visual (auditory /ta/ + visual /pa/) and acoustic deviants (auditory/pa/ + visual /ta/) were compared with frequent audiovisual standards (auditory /ta/ and visual /ta/). Statistical probability mapping revealed spectral amplitude increases at approximately 75 and approximately 78 Hz to visual deviants. GBA to visual deviants peaked 160 ms after auditory stimulus onset over posterior parietal cortex, at 270 ms over occipital areas and at 320 ms over left inferior frontal cortex. The latter GBA enhancement was consistent with the increase observed previously to pure acoustic mismatch, supporting a role of left inferior frontal cortex for the representation of perceived auditory pattern change. The preceding gamma-band changes over posterior areas may reflect processing of incongruent lip movements in visual motion areas and back-projections to earlier visual cortex.     

Cortical substrates supporting visual search in humans.

Serial and parallel visual search tasks were presented to patients with focal lesions in dorsolateral frontal, lateral parietal, or temporal-parietal cortex. In the unilateral display conditions, search efficiency in all patient groups was similar to the normal control group for stimuli both on the ipsi- and on the contralesional side of the displays. In contrast, in the bilateral display conditions, all patient groups showed a marked delay in initiating search on the side contralateral to the lesion as compared to normal controls. This delay was more pronounced when attention demands on the ipsilateral side increased, either by making target-distractor discrimination more difficult (serial search task), or by increasing the number of ipsilateral distractor items. The contralateral deficit was evident in all patient groups, supporting the notion that dorsolateral frontal as well as posterior parietal and temporal-parietal cortex plays a critical role in visual spatial attention.     

The role of the inferior parietal cortex in linking the tactile perception and manual construction of object shapes

We employed functional magnetic resonance imaging (fMRI) in 12 healthy subjects to measure cerebral activation related to a set of higher order manual sensorimotor tasks performed in the absence of visual guidance. Purposeless manipulation of meaningless plasticine lumps served as a reference against which we contrasted two tasks where manual manipulation served a meaningful purpose, either the perception and recognition of three-dimensional shapes or the construction of such shapes out of an amorphous plasticine lump. These tasks were compared with the corresponding mental imagery of the modelling process which evokes the constructive concept but lacks concomitant sensorimotor input and output. Neural overlap was found in a bilateral activity increase in the posterior and anterior intraparietal sulcus area (IPS and AIP). Differential activation was seen in the supplementary and cingulate motor areas, the left M1 and the superior parietal lobe for modelling and in the left angular and ventral premotor cortex for imagery. Our data thus point to a congruent neural substrate for both perceptive and constructive object-oriented sensorimotor cognition in the AIP and posterior IPS. The leftward asymmetry of the inferior parietal activations, including the angular gyrus, during imagery of modelling along with the ventral premotor activations emphasize the close vicinity of the circuitry for cognitive manipulative motor behaviour and language.     

The interaction of brain regions during visual search processing as revealed by transcranial magnetic stimulation

Although it has long been known that right posterior parietal cortex (PPC) has a role in certain visual search tasks, and human motion area V5 is involved in processing tasks requiring attention to motion, little is known about how these areas may interact during the processing of a task requiring the speciality of each. Using transcranial magnetic stimulation (TMS), this study first established the specialization of each area in the form of a double dissociation; TMS to right PPC disrupted processing of a color/form conjunction and TMS to V5 disrupted processing of a motion/form conjunction. The key finding of this study is, however, if TMS is used to disrupt processing of V5 at its critical time of activation during the motion/form conjunction task, concurrent disruption of right PPC now has a significant effect, where TMS at PPC alone does not. Our findings challenge the conventional interpretation of the role of right PPC in conjunction search and spatial attention.     

Que PASA? The posterior-anterior shift in aging

A consistent finding from functional neuroimaging studies of cognitive aging is an age-related reduction in occipital activity coupled with increased frontal activity. This posterior-anterior shift in aging (PASA) has been typically attributed to functional compensation. The present functional magnetic resonance imaging sought to 1) confirm that PASA reflects the effects of aging rather than differences in task difficulty; 2) test the compensation hypothesis; and 3) investigate whether PASA generalizes to deactivations. Young and older participants were scanned during episodic retrieval and visual perceptual tasks, and age-related changes in brain activity common to both tasks were identified. The study yielded 3 main findings. First, inconsistent with a difficulty account, the PASA pattern was found across task and confidence levels when matching performance among groups. Second, supporting the compensatory hypothesis, age-related increases in frontal activity were positively correlated with performance and negatively correlated with the age-related occipital decreases. Age-related increases and correlations with parietal activity were also found. Finally, supporting the generalizability of the PASA pattern to deactivations, aging reduced deactivations in posterior midline cortex but increased deactivations in medial frontal cortex. Taken together, these findings demonstrate the validity, function, and generalizability of PASA, as well as its importance for the cognitive neuroscience of aging.