Steady-state evoked potentials as an index of multisensory temporal binding

Temporal congruency promotes perceptual binding of multisensory inputs. Here, we used EEG frequency-tagging to track cortical activities elicited by auditory and visual inputs separately, in the form of steady-state evoked potentials (SS-EPs). We tested whether SS-EPs could reveal a dynamic coupling of cortical activities related to the binding of auditory and visual inputs conveying synchronous vs. non-synchronous temporal periodicities, or beats. The temporally congruent audiovisual condition elicited markedly enhanced auditory and visual SS-EPs, as compared to the incongruent condition. Furthermore, an increased inter-trial phase coherence of both SS-EPs was observed in that condition. Taken together, these observations indicate that temporal congruency enhances the processing of multisensory inputs at sensory-specific stages of cortical processing, possibly through a dynamic binding by synchrony of the elicited activities and/or improved dynamic attending. Moreover, we show that EEG frequency-tagging with SS-EPs constitutes an effective tool to explore the neural dynamics of multisensory integration in the human brain.     

Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition

The superior temporal sulcus (STS) is a region involved in audiovisual integration. In non-human primates, multisensory neurons in STS display inverse effectiveness. In two fMRI studies using multisensory tool and speech stimuli presented at parametrically varied levels of signal strength, we show that the pattern of neural activation in human STS is also inversely effective. Although multisensory tool-defined and speech-defined regions of interest were non-overlapping, the pattern of inverse effectiveness was the same for tools and speech across regions. The findings suggest that, even though there are sub-regions in STS that are speech-selective, the manner in which visual and auditory signals are integrated in multisensory STS is not specific to speech.     

Human brain activity associated with audiovisual perception and attention

Coherent perception of objects in our environment often requires perceptual integration of auditory and visual information. Recent behavioral data suggest that audiovisual integration depends on attention. The current study investigated the neural basis of audiovisual integration using 3-Tesla functional magnetic resonance imaging (fMRI) in 12 healthy volunteers during attention to auditory or visual features, or audiovisual feature combinations of abstract stimuli (simultaneous harmonic sounds and colored circles). Audiovisual attention was found to modulate activity in the same frontal, temporal, parietal and occipital cortical regions as auditory and visual attention. In addition, attention to audiovisual feature combinations produced stronger activity in the superior temporal cortices than attention to only auditory or visual features. These modality-specific areas might be involved in attention-dependent perceptual binding of synchronous auditory and visual events into coherent audiovisual objects. Furthermore, the modality-specific temporal auditory and occipital visual cortical areas showed attention-related modulations during both auditory and visual attention tasks. This result supports the proposal that attention to stimuli in one modality can spread to encompass synchronously presented stimuli in another modality.     

Neural processing of asynchronous audiovisual speech perception

The temporal synchrony of auditory and visual signals is known to affect the perception of an external event, yet it is unclear what neural mechanisms underlie the influence of temporal synchrony on perception. Using parametrically varied levels of stimulus asynchrony in combination with BOLD fMRI, we identified two anatomically distinct subregions of multisensory superior temporal cortex (mSTC) that showed qualitatively distinct BOLD activation patterns. A synchrony-defined subregion of mSTC (synchronous > asynchronous) responded only when auditory and visual stimuli were synchronous, whereas a bimodal subregion of mSTC (auditory > baseline and visual > baseline) showed significant activation to all presentations, but showed monotonically increasing activation with increasing levels of asynchrony. The presence of two distinct activation patterns suggests that the two subregions of mSTC may rely on different neural mechanisms to integrate audiovisual sensory signals. An additional whole-brain analysis revealed a network of regions responding more with synchronous than asynchronous speech, including right mSTC, and bilateral superior colliculus, fusiform gyrus, lateral occipital cortex, and extrastriate visual cortex. The spatial location of individual mSTC ROIs was much more variable in the left than right hemisphere, suggesting that individual differences may contribute to the right lateralization of mSTC in a group SPM. These findings suggest that bilateral mSTC is composed of distinct multisensory subregions that integrate audiovisual speech signals through qualitatively different mechanisms, and may be differentially sensitive to stimulus properties including, but not limited to, temporal synchrony.     

Multisensory interactions in early evoked brain activity follow the principle of inverse effectiveness

A major determinant of multisensory integration, derived from single-neuron studies in animals, is the principle of inverse effectiveness (IE), which describes the phenomenon whereby maximal multisensory response enhancements occur when the constituent unisensory stimuli are minimally effective in evoking responses. Human behavioral studies, which have shown that multisensory interactions are strongest when stimuli are low in intensity are in agreement with the IE principle, but the neurophysiologic basis for this finding is unknown. In this high-density electroencephalography (EEG) study, we examined effects of stimulus intensity on multisensory audiovisual processing in event-related potentials (ERPs) and response time (RT) facilitation in the bisensory redundant target effect (RTE). The RTE describes that RTs are faster for bisensory redundant targets than for the respective unisensory targets. Participants were presented with semantically meaningless unisensory auditory, unisensory visual and bisensory audiovisual stimuli of low, middle and high intensity, while they were instructed to make a speeded button response when a stimulus in either modality was presented. Behavioral data showed that the RTE exceeded predictions on the basis of probability summations of unisensory RTs, indicative of integrative multisensory processing, but only for low intensity stimuli. Paralleling this finding, multisensory interactions in short latency (40-60ms) ERPs with a left posterior and right anterior topography were found particularly for stimuli with low intensity. Our findings demonstrate that the IE principle is applicable to early multisensory processing in humans.     

Spatial and temporal factors during processing of audiovisual speech: a PET study

Speech perception can use not only auditory signals, but also visual information from seeing the speaker's mouth. The relative timing and relative location of auditory and visual inputs are both known to influence crossmodal integration psychologically, but previous imaging studies of audiovisual speech focused primarily on just temporal aspects. Here we used Positron Emission Tomography (PET) during audiovisual speech processing to study how temporal and spatial factors might jointly affect brain activations. In agreement with previous work, synchronous versus asynchronous audiovisual speech yielded increased activity in multisensory association areas (e.g., superior temporal sulcus [STS]), plus in some unimodal visual areas. Our orthogonal manipulation of relative stimulus position (auditory and visual stimuli presented at same location vs. opposite sides) and stimulus synchrony showed that (i) ventral occipital areas and superior temporal sulcus were unaffected by relative location; (ii) lateral and dorsal occipital areas were selectively activated for synchronous bimodal stimulation at the same external location; (iii) right inferior parietal lobule was activated for synchronous auditory and visual stimuli at different locations, that is, in the condition classically associated with the 'ventriloquism effect' (shift of perceived auditory position toward the visual location). Thus, different brain regions are involved in different aspects of audiovisual integration. While ventral areas appear more affected by audiovisual synchrony (which can influence speech identification), more dorsal areas appear to be associated with spatial multisensory interactions.     

Change-driven cortical activation in multisensory environments: An MEG study

The quick detection of dynamic changes in multisensory environments is essential to survive dangerous events and orient attention to informative events. Previous studies have identified multimodal cortical areas activated by changes of visual, auditory, and tactile stimuli. In the present study, we used magnetoencephalography (MEG) to examine time-varying cortical processes responsive to unexpected unimodal changes during continuous multisensory stimulation. The results showed that there were change-driven cortical responses in multimodal areas, such as the temporo-parietal junction and middle and inferior frontal gyri, regardless of the sensory modalities where the change occurred. These multimodal activations accompanied unimodal activations, both of which in general had some peaks within 300 ms after the changes. Thus, neural processes responsive to unimodal changes in the multisensory environment are distributed at different timing in these cortical areas.     

Neural restoration of degraded audiovisual speech

When speech is interrupted by noise, listeners often perceptually "fill-in" the degraded signal, giving an illusion of continuity and improving intelligibility. This phenomenon involves a neural process in which the auditory cortex (AC) response to onsets and offsets of acoustic interruptions is suppressed. Since meaningful visual cues behaviorally enhance this illusory filling-in, we hypothesized that during the illusion, lip movements congruent with acoustic speech should elicit a weaker AC response to interruptions relative to static (no movements) or incongruent visual speech. AC response to interruptions was measured as the power and inter-trial phase consistency of the auditory evoked theta band (4-8 Hz) activity of the electroencephalogram (EEG) and the N1 and P2 auditory evoked potentials (AEPs). A reduction in the N1 and P2 amplitudes and in theta phase-consistency reflected the perceptual illusion at the onset and/or offset of interruptions regardless of visual condition. These results suggest that the brain engages filling-in mechanisms throughout the interruption, which repairs degraded speech lasting up to ~250 ms following the onset of the degradation. Behaviorally, participants perceived speech continuity over longer interruptions for congruent compared to incongruent or static audiovisual streams. However, this specific behavioral profile was not mirrored in the neural markers of interest. We conclude that lip-reading enhances illusory perception of degraded speech not by altering the quality of the AC response, but by delaying it during degradations so that longer interruptions can be tolerated.     

Multisensory speech perception without the left superior temporal sulcus

Converging evidence suggests that the left superior temporal sulcus (STS) is a critical site for multisensory integration of auditory and visual information during speech perception. We report a patient, SJ, who suffered a stroke that damaged the left tempo-parietal area, resulting in mild anomic aphasia. Structural MRI showed complete destruction of the left middle and posterior STS, as well as damage to adjacent areas in the temporal and parietal lobes. Surprisingly, SJ demonstrated preserved multisensory integration measured with two independent tests. First, she perceived the McGurk effect, an illusion that requires integration of auditory and visual speech. Second, her perception of morphed audiovisual speech with ambiguous auditory or visual information was significantly influenced by the opposing modality. To understand the neural basis for this preserved multisensory integration, blood-oxygen level dependent functional magnetic resonance imaging (BOLD fMRI) was used to examine brain responses to audiovisual speech in SJ and 23 healthy age-matched controls. In controls, bilateral STS activity was observed. In SJ, no activity was observed in the damaged left STS but in the right STS, more cortex was active in SJ than in any of the normal controls. Further, the amplitude of the BOLD response in right STS response to McGurk stimuli was significantly greater in SJ than in controls. The simplest explanation of these results is a reorganization of SJ's cortical language networks such that the right STS now subserves multisensory integration of speech.     

Auditory-visual multisensory interactions attenuate subsequent visual responses in humans

Effects of multisensory interactions on how subsequent sensory inputs are processed remain poorly understood. We investigated whether multisensory interactions between rudimentary visual and auditory stimuli (flashes and beeps) affect later visual processing. A 2 x 3 design varied the number of flashes (1 or 2) with the number of beeps (0, 1, or 2) presented on each trial, such that '2F1B' refers to the presentation of 2 flashes with 1 beep. Beeps, when present, were synchronous with the first flash, and pairs of stimuli within a trial were separated by 52 ms ISI. Subjects indicated the number of flashes presented. Electrical neuroimaging of 128-channel event-related potentials assessed both the electric field strength and topography. Isolation of responses a visual stimulus that was preceded by a multisensory event was achieved by calculating the difference between the 2F1B and 1F1B conditions, and responses to a visual stimulus preceded by a unisensory event were isolated by calculating the difference between the 2F0B and 1F0B conditions (MUL and VIS, respectively). Comparison of MUL and VIS revealed that the treatment of visual information was significantly attenuated approximately 160 ms after the onset of the second flash when it was preceded by a multisensory event. Source estimations further indicated that this attenuation occurred within low-level visual cortices. Multisensory interactions are ongoing in low-level visual cortices and affect incoming sensory processing. These data provide evidence that multisensory interactions are not restricted in time and can dramatically influence the treatment of subsequent stimuli, opening new lines of multisensory research.     

Magnetoencephalographic correlates of audiotactile interaction

To seek for correlates of an interaction between auditory and somatosensory processing, the brain's magnetic field in response to simultaneously presented auditory (A) and tactile (T) stimuli was compared with the sum of the respective unimodal responses (A+T). The stimuli were binaural 1047-Hz tone bursts of 60 dB sensation level and tactile pressure pulses to the right thumb. The mean interval between two stimuli of the same modality was 1.95 s. The magnetic field was recorded using a 306-channel whole-scalp neuromagnetometer. A clear audiotactile interaction was revealed in the hemisphere contralateral to the side of tactile stimulation in six of eight subjects, whereas in the ipsilateral hemisphere an interaction was noticed in only three subjects. The time courses of these audiotactile interaction fields typically showed major deflections of opposite polarities around 140 and 220 ms. The first deflection appeared to arise in the region of the secondary somatosensory cortex (SII). The polarity of this interaction was consistent with the view that the auditory stimulus resulted in a partial inhibition in SII. In two subjects, strong indications of auditory contributions to the interaction were available, although in different hemispheres. The relatively high interindividual variability of the observed interaction, which represents potential neural substrates for multisensory integration, could indicate that the way subjects perceive the simultaneous presentation of auditory and tactile stimuli differs.     

Brain activation during audiovisual exposure anticipates future perception of ambiguous speech

In modern perceptual neuroscience, the focus of interest has shifted from a restriction to individual modalities to an acknowledgement of the importance of multisensory processing. One particularly well-known example of cross-modal interaction is the McGurk illusion. It has been shown that this illusion can be modified, such that it creates an auditory perceptual bias that lasts beyond the duration of audiovisual stimulation, a process referred to as cross-modal recalibration (Bertelson et al., 2003). Recently, we have suggested that this perceptual bias is stored in auditory cortex, by demonstrating the feasibility of retrieving the subjective perceptual interpretation of recalibrated ambiguous phonemes from functional magnetic resonance imaging (fMRI) measurements in these regions (Kilian-Hütten et al., 2011). However, this does not explain which brain areas integrate the information from the two senses and represent the origin of the auditory perceptual bias. Here we analyzed fMRI data from audiovisual recalibration blocks, utilizing behavioral data from perceptual classifications of ambiguous auditory phonemes that followed these blocks later in time. Adhering to this logic, we could identify a network of brain areas (bilateral inferior parietal lobe [IPL], inferior frontal sulcus [IFS], and posterior middle temporal gyrus [MTG]), whose activation during audiovisual exposure anticipated auditory perceptual tendencies later in time. We propose a model in which a higher-order network, including IPL and IFS, accommodates audiovisual integrative learning processes, which are responsible for the installation of a perceptual bias in auditory regions. This bias then determines constructive perceptual processing.     

Audiovisual integration of emotional signals in voice and face: an event-related fMRI study

In a natural environment, non-verbal emotional communication is multimodal (i.e. speech melody, facial expression) and multifaceted concerning the variety of expressed emotions. Understanding these communicative signals and integrating them into a common percept is paramount to successful social behaviour. While many previous studies have focused on the neurobiology of emotional communication in the auditory or visual modality alone, far less is known about multimodal integration of auditory and visual non-verbal emotional information. The present study investigated this process using event-related fMRI. Behavioural data revealed that audiovisual presentation of non-verbal emotional information resulted in a significant increase in correctly classified stimuli when compared with visual and auditory stimulation. This behavioural gain was paralleled by enhanced activation in bilateral posterior superior temporal gyrus (pSTG) and right thalamus, when contrasting audiovisual to auditory and visual conditions. Further, a characteristic of these brain regions, substantiating their role in the emotional integration process, is a linear relationship between the gain in classification accuracy and the strength of the BOLD response during the bimodal condition. Additionally, enhanced effective connectivity between audiovisual integration areas and associative auditory and visual cortices was observed during audiovisual stimulation, offering further insight into the neural process accomplishing multimodal integration. Finally, we were able to document an enhanced sensitivity of the putative integration sites to stimuli with emotional non-verbal content as compared to neutral stimuli.     

Spatiotemporal dynamics of audiovisual speech processing

The cortical processing of auditory-alone, visual-alone, and audiovisual speech information is temporally and spatially distributed, and functional magnetic resonance imaging (fMRI) cannot adequately resolve its temporal dynamics. In order to investigate a hypothesized spatiotemporal organization for audiovisual speech processing circuits, event-related potentials (ERPs) were recorded using electroencephalography (EEG). Stimuli were congruent audiovisual/ba/, incongruent auditory/ba/synchronized with visual/ga/, auditory-only/ba/, and visual-only/ba/and/ga/. Current density reconstructions (CDRs) of the ERP data were computed across the latency interval of 50-250 ms. The CDRs demonstrated complex spatiotemporal activation patterns that differed across stimulus conditions. The hypothesized circuit that was investigated here comprised initial integration of audiovisual speech by the middle superior temporal sulcus (STS), followed by recruitment of the intraparietal sulcus (IPS), followed by activation of Broca's area [Miller, L.M., d'Esposito, M., 2005. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. Journal of Neuroscience 25, 5884-5893]. The importance of spatiotemporally sensitive measures in evaluating processing pathways was demonstrated. Results showed, strikingly, early (<100 ms) and simultaneous activations in areas of the supramarginal and angular gyrus (SMG/AG), the IPS, the inferior frontal gyrus, and the dorsolateral prefrontal cortex. Also, emergent left hemisphere SMG/AG activation, not predicted based on the unisensory stimulus conditions was observed at approximately 160 to 220 ms. The STS was neither the earliest nor most prominent activation site, although it is frequently considered the sine qua non of audiovisual speech integration. As discussed here, the relatively late activity of the SMG/AG solely under audiovisual conditions is a possible candidate audiovisual speech integration response.     

Mind's ear in a musician: where and when in the brain

The temporospatial pattern of brain activity during auditory imagery was studied using magnetoencephalography. Trained musicians were presented with visual notes and instructed to imagine the corresponding sounds. Brain activity specific to the auditory imagery task was observed, first as enhanced activity of left and right occipital areas (average onset 120-150 ms after the onset of the visual stimulus) and then spreading to the midline parietal cortex (precuneus) and to such extraoccipital areas that were not activated during the visual control condition (e.g., the left temporal auditory association cortex and the left and right premotor cortices). The latest activations, with average onset latencies of 270-400 ms clearly separate from the earliest ones, occurred in the left sensorimotor cortex and the right inferotemporal visual association cortex. These data imply a complex temporospatial activation sequence of multiple cortical areas when musicians recall firmly established audiovisual associations.     

The temporal dynamics of processing emotions from vocal, facial, and bodily expressions

Face-to-face communication works multimodally. Not only do we employ vocal and facial expressions; body language provides valuable information as well. Here we focused on multimodal perception of emotion expressions, monitoring the temporal unfolding of the interaction of different modalities in the electroencephalogram (EEG). In the auditory condition, participants listened to emotional interjections such as "ah", while they saw mute video clips containing emotional body language in the visual condition. In the audiovisual condition participants saw video clips with matching interjections. In all three conditions, the emotions "anger" and "fear", as well as non-emotional stimuli were used. The N100 amplitude was strongly reduced in the audiovisual compared to the auditory condition, suggesting a significant impact of visual information on early auditory processing. Furthermore, anger and fear expressions were distinct in the auditory but not the audiovisual condition. Complementing these event-related potential (ERP) findings, we report strong similarities in the alpha- and beta-band in the visual and the audiovisual conditions, suggesting a strong visual processing component in the perception of audiovisual stimuli. Overall, our results show an early interaction of modalities in emotional face-to-face communication using complex and highly natural stimuli.     

Sound alters activity in human V1 in association with illusory visual perception

When a single brief visual flash is accompanied by two auditory bleeps, it is frequently perceived incorrectly as two flashes. Here, we used high field functional MRI in humans to examine the neural basis of this multisensory perceptual illusion. We show that activity in retinotopic visual cortex is increased by the presence of concurrent auditory stimulation, irrespective of any illusory perception. However, when concurrent auditory stimulation gave rise to illusory visual perception, activity in V1 was enhanced, despite auditory and visual stimulation being unchanged. These findings confirm that responses in human V1 can be altered by sound and show that they reflect subjective perception rather than the physically present visual stimulus. Moreover, as the right superior temporal sulcus and superior colliculus were also activated by illusory visual perception, together with V1, they provide a potential neural substrate for the generation of this multisensory illusion.     

Plasticity in representations of environmental sounds revealed by electrical neuroimaging

The rapid and precise processing of environmental sounds contributes to communication functions as well as both object recognition and localization. Plasticity in (accessing) the neural representations of environmental sounds is likewise essential for an adaptive organism, in particular humans, and can be indexed by repetition priming. How the brain achieves such plasticity with representations of environmental sounds is presently unresolved. Electrical neuroimaging of 64-channel auditory evoked potentials (AEPs) in humans identified the spatio-temporal brain mechanisms of repetition priming involving sounds of environmental objects. Subjects performed an 'oddball' target detection task, based on the semantic category of stimuli (living vs. man-made objects). Repetition priming effects were observed behaviorally as a speeding of reaction times and electrophysiologically as a suppression of the strength of responses to repeated sound presentations over the 156-215 ms post-stimulus period. These effects of plasticity were furthermore localized, using statistical analyses of a distributed linear inverse solution, to the left middle temporal gyrus and superior temporal sulcus (BA22), which have been implicated in associating sounds with their abstract representations and actions. These effects are subsequent to and occur in different brain regions from what has been previously identified as the earliest discrimination of auditory object categories. Plasticity in associative-semantic, rather than perceptual-discriminative functions, may underlie repetition priming of sounds of objects. We present a multi-stage mechanism of auditory object processing akin to what has been described for visual object processing and which also provides a framework for accessing multisensory object representations.     

Gamma-band activity over early sensory areas predicts detection of changes in audiovisual speech stimuli

Oscillatory activity in the gamma-band range in human magneto- and electroencephalogram is thought to reflect the oscillatory synchronization of cortical networks. Findings of enhanced gamma-band activity (GBA) during cognitive processes like gestalt perception, attention and memory have led to the notion that GBA may reflect the activation of internal object representations. However, there is little direct evidence suggesting that GBA is related to subjective perceptual experience. In the present study, magnetoencephalogram was recorded during an audiovisual oddball paradigm with infrequent visual (auditory /ta/ + visual /pa/) or acoustic deviants (auditory /pa/ + visual /ta/) interspersed in a sequence of frequent audiovisual standard stimuli (auditory /ta/ + visual /ta/). Sixteen human subjects had to respond to perceived acoustic changes which could be produced either by real acoustic or illusory (visual) deviants. Statistical probability mapping served to identify correlations between oscillatory activity in response to visual and acoustic deviants, respectively, and the detection rates for either type of deviant. The perception of illusory acoustic changes induced by visual deviants was closely associated with gamma-band amplitude at approximately 80 Hz between 250 and 350 ms over midline occipital cortex. In contrast, the detection of real acoustic deviants correlated positively with induced GBA at approximately 42 Hz between 200 and 300 ms over left superior temporal cortex and negatively with evoked gamma responses at approximately 41 Hz between 220 and 240 ms over occipital areas. These findings support the relevance of high-frequency oscillatory activity over early sensory areas for perceptual experience.     

Touch, sound and vision in human superior temporal sulcus

Human superior temporal sulcus (STS) is thought to be a key brain area for multisensory integration. Many neuroimaging studies have reported integration of auditory and visual information in STS but less is known about the role of STS in integrating other sensory modalities. In macaque STS, the superior temporal polysensory area (STP) responds to somatosensory, auditory and visual stimulation. To determine if human STS contains a similar area, we measured brain responses to somatosensory, auditory and visual stimuli using blood-oxygen level dependent functional magnetic resonance imaging (BOLD fMRI). An area in human posterior STS, STSms (multisensory), responded to stimulation in all three modalities. STSms responded during both active and passive presentation of unisensory somatosensory stimuli and showed larger responses for more intense vs. less intense tactile stimuli, hand vs. foot, and contralateral vs. ipsilateral tactile stimulation. STSms showed responses of similar magnitude for unisensory tactile and auditory stimulation, with an enhanced response to simultaneous auditory-tactile stimulation. We conclude that STSms is important for integrating information from the somatosensory as well as the auditory and visual modalities, and could be the human homolog of macaque STP.