Hearing lips and seeing voices: how cortical areas supporting speech production mediate audiovisual speech perception

Observing a speaker's mouth profoundly influences speech perception. For example, listeners perceive an "illusory" "ta" when the video of a face producing /ka/ is dubbed onto an audio /pa/. Here, we show how cortical areas supporting speech production mediate this illusory percept and audiovisual (AV) speech perception more generally. Specifically, cortical activity during AV speech perception occurs in many of the same areas that are active during speech production. We find that different perceptions of the same syllable and the perception of different syllables are associated with different distributions of activity in frontal motor areas involved in speech production. Activity patterns in these frontal motor areas resulting from the illusory "ta" percept are more similar to the activity patterns evoked by AV(/ta/) than they are to patterns evoked by AV(/pa/) or AV(/ka/). In contrast to the activity in frontal motor areas, stimulus-evoked activity for the illusory "ta" in auditory and somatosensory areas and visual areas initially resembles activity evoked by AV(/pa/) and AV(/ka/), respectively. Ultimately, though, activity in these regions comes to resemble activity evoked by AV(/ta/). Together, these results suggest that AV speech elicits in the listener a motor plan for the production of the phoneme that the speaker might have been attempting to produce, and that feedback in the form of efference copy from the motor system ultimately influences the phonetic interpretation.     

Lateralization of dichotic speech stimuli is based on specific auditory pathway interactions: neuromagnetic evidence

Dichotic listening (DL) is a neuropsychological technique for the study of functional laterality. Based on behavioral patient studies, the "structural theory" states that lateralization of the auditory input during DL is allowed by an inhibition of the ipsilateral pathways. We aimed here at extending this theory to provide a neurophysiological basis of verbal DL. We investigated the magnetic responses of the primary auditory cortices elicited by dichotic consonant-vowel syllables. Dichotic stimuli consisted of 2 syllables pairs, a "competing" one composed by syllables with high spectral overlap (/da/ and /ba/) and a "noncompeting" pair (/da/ and /ka/). One of the syllables in each pair was delivered at 2 intensities, whereas the other did not change. A reduced increase of source intensity in response to dichotic pairs at the 2 levels was assumed to indicate pathway inhibition effects. We obtained that the left ipsilateral pathway (i.e., the left ipsilateral signal) was strongly inhibited by the right contralateral one. Conversely, the right ipsilateral pathway did not show an inhibition larger than the left contralateral one. These results extend the notion of auditory functional asymmetries by showing that beyond hemispheric functional specialization there is an asymmetry within the ascending auditory system, which is based on a competition mechanism. The larger the competition between the left and right ear stimuli, the larger are the inhibition effects, which determine the pathway asymmetry. These findings represent as well a neurophysiological basis for the "structural theory" explaining the right ear preference usually found in verbal DL tasks.     

Binding symbols and sounds: evidence from event-related oscillatory gamma-band activity

The present study intended to examine the neural basis of audiovisual integration, hypothetically achieved by synchronized gamma-band oscillations (30-80 Hz) that have been suggested to integrate stimulus features and top-down information. To that end, we studied the impact of visual symbolic information on early auditory sensory processing of upcoming sounds. In particular, we used a symbol-to-sound-matching paradigm in which simple score-like patterns predict corresponding sound patterns. Occasionally, a single sound is incongruent with the corresponding element of the visual pattern. In response to expected sounds congruent with the corresponding visual symbol, a power increase of phase-locked (evoked) activity in the 40-Hz band was observed peaking 42-ms poststimulus onset. Thus, for the first time, we demonstrated that the comparison process between a neural model, the expectation, and the current sensory input is implemented at very early levels of auditory processing. Subsequently, expected congruent sounds elicited a broadband power increase of non-phase-locked (induced) activity peaking 152-ms poststimulus onset, which might reflect the formation of a unitary event representation including both visual and auditory aspects of the stimulation. Gamma-band responses were not present for unexpected incongruent sounds. A model explaining the anticipatory activation of cortical auditory representations and the match of experience against expectation is presented.     

Human auditory cortical dynamics during perception of long acoustic sequences: phase tracking of carrier frequency by the auditory steady-state response

We recorded human auditory cortical activity during the perception of long, changing acoustic signals and analyzed information provided by dynamic neural population measures over a large range of time intervals (approximately 24 ms-5 s). Participants listened to musical scales that were amplitude modulated at a rate of 41.5 Hz, generating an ongoing, stimulus-related oscillatory brain signal, the auditory steady-state response (aSSR). The aSSR generated energy at the amplitude modulation rate that was recorded using magnetoencephalography. As in previous work, the timing (phase) of this response varied with stimulus carrier frequency over the entire course of minute-long tone sequences ('phase tracking' of carrier frequency). The length of the time interval over which phase was calculated was systematically varied; significant phase tracking was regularly observed at analysis intervals of <50 ms in length. The right auditory cortex exhibited better phase tracking performance than the left at analysis intervals of 24-240 ms, and frequency dependent phase delays were consistently larger than those predicted by cochlear mechanics. Based on these empirical data, a model of the neural populations responsible for phase tracking suggests that it is produced by a subpopulation ( approximately 25%) of the cells generating the aSSR.     

Dynamics of oscillatory activity during auditory decision making

Perceptual decision making requires the comparison and integration of sensory evidence to generate a behavioral response. We used magnetoencephalography to investigate the temporal dynamics of decision making during an auditory task that required forced-choice decisions about whether a pair of syllables S1 and S2 differed either in their acoustic patterns or in the perceived position of their sound sources. Conditions with easy and difficult decisions were created by varying the similarity of S1 and S2. Statistical probability mapping showed enhanced gamma-band activity (GBA) over posterior parietal cortex for spatial and over left inferior frontal cortex for pattern changes (at approximately 120 to 220 ms after S2 onset). Activations were stronger for easy than difficult decisions. GBA over dorsolateral prefrontal cortex was more pronounced at approximately 280 to 430 ms for easy than difficult decisions regardless of type of change, possibly reflecting decision-relevant networks that integrate information from higher sensory areas representing the perceptual alternatives. Sensorimotor beta desynchronization as a measure of motor preparation peaked at approximately 460 ms for easy and at approximately 520 ms for difficult decisions, thus reflecting the reaction time difference between both conditions. In summary, the analysis of oscillatory activity in magnetoencephalogram served to elucidate the temporal dynamics of perceptual decision making in humans.