Differential Auditory and Visual Phase-Locking Are Observed during Audio-Visual Benefit and Silent Lip-Reading for Speech Perception

Speech perception in noisy environments is enhanced by seeing facial movements of communication partners. However, the neural mechanisms by which audio and visual speech are combined are not fully understood. We explore MEG phase-locking to auditory and visual signals in MEG recordings from 14 human participants (6 females, 8 males) that reported words from single spoken sentences. We manipulated the acoustic clarity and visual speech signals such that critical speech information is present in auditory, visual, or both modalities. MEG coherence analysis revealed that both auditory and visual speech envelopes (auditory amplitude modulations and lip aperture changes) were phase-locked to 2-6 Hz brain responses in auditory and visual cortex, consistent with entrainment to syllable-rate components. Partial coherence analysis was used to separate neural responses to correlated audio-visual signals and showed non-zero phase-locking to auditory envelope in occipital cortex during audio-visual (AV) speech. Furthermore, phase-locking to auditory signals in visual cortex was enhanced for AV speech compared with audio-only speech that was matched for intelligibility. Conversely, auditory regions of the superior temporal gyrus did not show above-chance partial coherence with visual speech signals during AV conditions but did show partial coherence in visual-only conditions. Hence, visual speech enabled stronger phase-locking to auditory signals in visual areas, whereas phase-locking of visual speech in auditory regions only occurred during silent lip-reading. Differences in these cross-modal interactions between auditory and visual speech signals are interpreted in line with cross-modal predictive mechanisms during speech perception.SIGNIFICANCE STATEMENT Verbal communication in noisy environments is challenging, especially for hearing-impaired individuals. Seeing facial movements of communication partners improves speech perception when auditory signals are degraded or absent. The neural mechanisms supporting lip-reading or audio-visual benefit are not fully understood. Using MEG recordings and partial coherence analysis, we show that speech information is used differently in brain regions that respond to auditory and visual speech. While visual areas use visual speech to improve phase-locking to auditory speech signals, auditory areas do not show phase-locking to visual speech unless auditory speech is absent and visual speech is used to substitute for missing auditory signals. These findings highlight brain processes that combine visual and auditory signals to support speech understanding.     

Audio-visual matching of speech and non-speech oral gestures in patients with aphasia and apraxia of speech

BACKGROUND: Audio-visual speech perception mechanisms provide evidence for a supra-modal nature of phonological representations, and a link of these mechanisms to motor representations of speech has been postulated. This leads to the question if aphasic patients and patients with apraxia of speech are able to exploit the visual signal in speech perception and if implicit knowledge of audio-visual relationships is preserved in these patients. Moreover, it is unknown if the audio-visual processing of mouth movements has a specific organisation in the speech as compared to the non-speech domain. METHODS: A discrimination task with speech and non-speech stimuli was applied in four presentation modes: auditory, visual, bimodal and cross-modal. We investigated 14 healthy persons and 14 patients with aphasia and/or apraxia of speech. RESULTS: Patients made substantially more errors than normal subjects on both the speech and the non-speech stimuli, in all presentation modalities. Normal controls made only few errors on the speech stimuli, regardless of the presentation mode, but had a high between-subject variability in the cross-modal matching of non-speech stimuli. The patients' cross-modal processing of non-speech stimuli was mainly predicted by lower face apraxia scores, while their audio-visual matching of syllables was predicted by word repetition abilities and the presence of apraxia of speech. CONCLUSIONS: (1) Impaired speech perception in aphasia is located at a supra-modal representational level. (2) Audio-visual processing is different for speech and non-speech oral gestures. (3) Audio-visual matching abilities in patients with left-hemisphere lesions depend on their speech and non-speech motor abilities.     

Perception of the auditory-visual illusion in speech perception by children with phonological disorders

An example of the auditory-visual illusion in speech perception, first described by McGurk and MacDonald, is the perception of [ta] when listeners hear [pa] in synchrony with the lip movements for [ka]. One account of the illusion is that lip-read and heard speech are combined in an articulatory code since people who mispronounce words respond differently from controls on lip-reading tasks. A same-different judgment task assessing perception of the illusion showed no difference in performance between controls and children with speech difficulties. Another experiment compared children with delayed and disordered speech on perception of the illusion. While neither group perceived many illusions, a significant interaction indicated that children with disordered phonology were strongly biased to the auditory component while the delayed group's response was more evenly split between the auditory and visual components of the illusion. These findings suggest that phonological processing, rather than articulation, supports lip-reading ability.     

Evolution of non-speech sound memory in postlingual deafness: implications for cochlear implant rehabilitation

Neurofunctional patterns assessed before or after cochlear implantation (CI) are informative markers of implantation outcome. Because phonological memory reorganization in post-lingual deafness is predictive of the outcome, we investigated, using a cross-sectional approach, whether memory of non-speech sounds (NSS) produced by animals or objects (i.e. non-human sounds) is also reorganized, and how this relates to speech perception after CI. We used an fMRI auditory imagery task in which sounds were evoked by pictures of noisy items for post-lingual deaf candidates for CI and for normal-hearing subjects. When deaf subjects imagined sounds, the left inferior frontal gyrus, the right posterior temporal gyrus and the right amygdala were less activated compared to controls. Activity levels in these regions decreased with duration of auditory deprivation, indicating declining NSS representations. Whole brain correlations with duration of auditory deprivation and with speech scores after CI showed an activity decline in dorsal, fronto-parietal, cortical regions, and an activity increase in ventral cortical regions, the right anterior temporal pole and the hippocampal gyrus. Both dorsal and ventral reorganizations predicted poor speech perception outcome after CI. These results suggest that post-CI speech perception relies, at least partially, on the integrity of a neural system used for processing NSS that is based on audio-visual and articulatory mapping processes. When this neural system is reorganized, post-lingual deaf subjects resort to inefficient semantic- and memory-based strategies. These results complement those of other studies on speech processing, suggesting that both speech and NSS representations need to be maintained during deafness to ensure the success of CI.     

Mu wave suppression during the perception of meaningless syllables: EEG evidence of motor recruitment

Motor involvement in speech perception has been recently studied using a variety of techniques. In the current study, EEG measurements from Cz, C3 and C4 electrodes were used to examine the relative power of the mu rhythm (i.e., 8–13 Hz) in response to various audio-visual speech and non-speech stimuli, as suppression of these rhythms is considered an index of ‘mirror neuron’ (i.e., motor) activity. Fourteen adult native English speaking females watched and listened to nine audio-video stimuli clips assembled from three different auditory stimuli (speech, noise, and pure tone) combined with three different video stimuli (speech, noise, and kaleidoscope—made from scrambling an image from the visual speech). Relative to the noise–noise (baseline condition), all visual speech conditions resulted in significant levels of suppression, a finding that is consistent with previous reports of mirror activity to visual speech and mu suppression to ‘biological’ stimuli. None of the non-speech conditions or conditions in which speech was presented via audition only resulted in any significant suppression of the mu rhythm in this population. Thus, visual speech perception appears to be more closely associated with motor activity than acoustic speech perception. It is postulated that in this study, the processing demands incurred by the task were insufficient for inducing significant mu suppression via acoustic speech only. The findings are discussed in theoretical contexts of speech perception and the mirror system. We suggest that this technique may offer a cost-efficient, non-invasive technique for measuring motor activity during speech perception.     

Sequential audiovisual interactions during speech perception: a whole-head MEG study

Using whole-head magnetoencephalography (MEG), audiovisual (AV) interactions during speech perception (/ta/- and /pa/-syllables) were investigated in 20 subjects. Congruent AV events served as the 'standards' of an oddball design. The deviants encompassed incongruent /ta/-/pa/ configurations differing from the standards either in the acoustic or the visual domain. As an auditory non-speech control condition, the same video signals were synchronized with either one of two complex tones. As in natural speech, visual movement onset preceded acoustic signals by about 150 ms. First, the impact of visual information on auditorily evoked fields to non-speech sounds was determined. Larger facial movements (/pa/ versus /ta/) yielded enhanced early responses such as the M100 component, indicating, most presumably, anticipatory pre-activation of auditory cortex by visual motion cues. As a second step of analysis, mismatch fields (MMF) were calculated. Acoustic deviants elicited a typical MMF, peaking ca. 180 ms after stimulus onset, whereas visual deviants gave rise to later responses (220 ms) of a more posterior-medial source location. Finally, a late (275 ms), left-lateralized visually-induced MMF component, resembling the acoustic mismatch response, emerged during the speech condition, presumably reflecting phonetic/linguistic operations. There is mounting functional imaging evidence for an early impact of visual information on auditory cortical regions during speech perception. The present study suggests at least two successive AV interactions in association with syllable recognition tasks: early activation of auditory areas depending upon visual motion cues and a later speech-specific left-lateralized response mediated, conceivably, by backward-projections from multisensory areas.     

Adaptation of neuromagnetic N1 responses to phonetic stimuli by visual speech in humans

The technique of 306-channel magnetoencephalogaphy (MEG) was used in eight healthy volunteers to test whether silent lip-reading modulates auditory-cortex processing of phonetic sounds. Auditory test stimuli (either Finnish vowel /ae/ or /?/) were preceded by a 500 ms lag by either another auditory stimulus (/ae/, /?/ or the second-formant midpoint between /ae/ and /?/), or silent movie of a person articulating /ae/ or /?/. Compared with N1 responses to auditory /ae/ and /?/ when presented without a preceding stimulus, the amplitudes of left-hemisphere N1 responses to the test stimuli were significantly suppressed both when preceded by auditory and visual stimuli, this effect being significantly stronger with preceding auditory stimuli. This suggests that seeing articulatory gestures of a speaker influences auditory speech perception by modulating the responsiveness of auditory-cortex neurons.     

Visual speech perception without primary auditory cortex activation

Speech perception is conventionally thought to be an auditory function, but humans often use their eyes to perceive speech. We investigated whether visual speech perception depends on processing by the primary auditory cortex in hearing adults. In a functional magnetic resonance imaging experiment, a pulse-tone was presented contrasted with gradient noise. During the same session, a silent video of a talker saying isolated words was presented contrasted with a still face. Visual speech activated the superior temporal gyrus anterior, posterior, and lateral to the primary auditory cortex, but not the region of the primary auditory cortex. These results suggest that visual speech perception is not critically dependent on the region of primary auditory cortex.     

Response amplification in sensory-specific cortices during crossmodal binding

Integrating information across the senses can enhance our ability to detect and classify stimuli in the environment. For example, auditory speech perception is substantially improved when the speaker's face is visible. In an fMRI study designed to investigate the neural mechanisms underlying these crossmodal behavioural gains, bimodal (audio-visual) speech was contrasted against both unimodal (auditory and visual) components. Significant response enhancements in auditory (BA 41/42) and visual (V5) cortices were detected during bimodal stimulation. This effect was found to be specific to semantically congruent crossmodal inputs. These data suggest that the perceptual improvements effected by synthesizing matched multisensory inputs are realised by reciprocal amplification of the signal intensity in participating unimodal cortices.     

A case of impaired auditory and visual speech prosody perception after right hemisphere damage

It is well established that vision plays a role in segmental speech perception, but the role of vision in prosodic speech perception is less clear. We report on the difficulties in prosodic speech perception encountered by KB after a right hemisphere stroke. In addition to musical deficits, KB was suspected of having impaired auditory prosody perception. As expected, KB was impaired on two prosody perception tasks in an auditory-only condition. We also examined whether the addition of visual prosody cues would facilitate his performance on these tasks. Unexpectedly, KB was also impaired on both tasks under visual-only and audio-visual conditions. Thus, there was no evidence that KB could integrate auditory and visual prosody information or that he could use visual cues to compensate for his deficit in the auditory domain. In contrast, KB was able to identify segmental speech information using visual cues and to use these visual cues to improve his performance when auditory segmental cues were impoverished. KB was also able to integrate audio-visual segmental information in the McGurk effect. Thus, KB's visual deficit was specific to prosodic speech perception and, to our knowledge, this is the first reported case of such a deficit.     

A Case of Impaired Auditory and Visual Speech Prosody Perception after Right Hemisphere Damage

It is well established that vision plays a role in segmental speech perception, but the role of vision in prosodic speech perception is less clear. We report on the difficulties in prosodic speech perception encountered by KB after a right hemisphere stroke. In addition to musical deficits, KB was suspected of having impaired auditory prosody perception. As expected, KB was impaired on two prosody perception tasks in an auditory-only condition. We also examined whether the addition of visual prosody cues would facilitate his performance on these tasks. Unexpectedly, KB was also impaired on both tasks under visual-only and audio-visual conditions. Thus, there was no evidence that KB could integrate auditory and visual prosody information or that he could use visual cues to compensate for his deficit in the auditory domain. In contrast, KB was able to identify segmental speech information using visual cues and to use these visual cues to improve his performance when auditory segmental cues were impoverished. KB was also able to integrate audio-visual segmental information in the McGurk effect. Thus, KB’s visual deficit was specific to prosodic speech perception and, to our knowledge, this is the first reported case of such a deficit.     

Speechreading circuits in people born deaf

In hearing people, silent speechreading generates bilateral activation in superior temporal regions specialised for the perception of auditory speech [Science 276 (1997) 593; Neuroreport 11 (2000) 1729; Proceedings of the Royal Society London B 268 (2001) 451]. In the present study, FMRI data were collected from deaf and hearing volunteers while they speechread numbers and during a control task in which they counted nonsense mouth movements (gurns). Brain activation for silent speechreading in oral deaf participants was found primarily in posterior cingulate cortex and hippocampal/lingual gyri. In contrast to the pattern observed in the hearing group, deaf participants showed no speechreading-specific activation in left lateral temporal regions.These data suggest that acoustic experience shapes the functional circuits for analysing speech. We speculate on the functional role, the posterior cingulate gyrus may play in speechreading by profoundly congenitally deaf people.     

The contribution of visual areas to speech comprehension: a PET study in cochlear implants patients and normal-hearing subjects

Early visual cortex can be recruited by meaningful sounds in the absence of visual information. This occurs in particular in cochlear implant (CI) patients whose dependency on visual cues in speech comprehension is increased. Such cross-modal interaction mirrors the response of early auditory cortex to mouth movements (speech reading) and may reflect the natural expectancy of the visual counterpart of sounds, lip movements. Here we pursue the hypothesis that visual activations occur specifically in response to meaningful sounds. We performed PET in both CI patients and controls, while subjects listened either to their native language or to a completely unknown language. A recruitment of early visual cortex, the left posterior inferior temporal gyrus (ITG) and the left superior parietal cortex was observed in both groups. While no further activation occurred in the group of normal-hearing subjects, CI patients additionally recruited the right perirhinal/fusiform and mid-fusiform, the right temporo-occipito-parietal (TOP) junction and the left inferior prefrontal cortex (LIPF, Broca's area). This study confirms a participation of visual cortical areas in semantic processing of speech sounds. Observation of early visual activation in normal-hearing subjects shows that auditory-to-visual cross-modal effects can also be recruited under natural hearing conditions. In cochlear implant patients, speech activates the mid-fusiform gyrus in the vicinity of the so-called face area. This suggests that specific cross-modal interaction involving advanced stages in the visual processing hierarchy develops after cochlear implantation and may be the correlate of increased usage of lip-reading.     

Audio-visual integration in schizophrenia

Integration of information provided simultaneously by audition and vision was studied in a group of 18 schizophrenic patients. They were compared to a control group, consisting of 12 normal adults of comparable age and education. By administering two tasks, each focusing on one aspect of audio-visual integration, the study could differentiate between a spatial integration deficit and a speech-based integration deficit. Experiment 1 studied audio-visual interactions in the spatial localisation of sounds. Experiment 2 investigated integration of auditory and visual speech. The schizophrenic group performed as the control group on the sound localisation task, but in the audio-visual speech task, there was an impairment in lipreading as well as a smaller impact of lipreading on auditory speech information. Combined with findings about functional and neuro-anatomical specificity of intersensory integration, the data suggest that there is an integration deficit in the schizophrenic group that is related to the processing of phonetic information.     

Bimodal speech: early suppressive visual effects in human auditory cortex

While everyone has experienced that seeing lip movements may improve speech perception, little is known about the neural mechanisms by which audiovisual speech information is combined. Event-related potentials (ERPs) were recorded while subjects performed an auditory recognition task among four different natural syllables randomly presented in the auditory (A), visual (V) or congruent bimodal (AV) condition. We found that: (i) bimodal syllables were identified more rapidly than auditory alone stimuli; (ii) this behavioural facilitation was associated with cross-modal [AV-(A+V)] ERP effects around 120-190 ms latency, expressed mainly as a decrease of unimodal N1 generator activities in the auditory cortex. This finding provides evidence for suppressive, speech-specific audiovisual integration mechanisms, which are likely to be related to the dominance of the auditory modality for speech perception. Furthermore, the latency of the effect indicates that integration operates at pre-representational stages of stimulus analysis, probably via feedback projections from visual and/or polymodal areas.     

Neural intersections of the phonological,visual magnocellular and motor/cerebellar systems in normal readers: Implications for imaging studies on dyslexia

We used fMRI to explore the extent of the anatomical overlap of three neural systems that the literature on developmental dyslexia associates with reading: the auditory phonological, the visual magnocellular, and the motor/cerebellar systems. Twenty‐eight normal subjects performed four tasks during fMRI scans: word and pseudoword reading, auditory rhyming for letter names, visual motion perception, and a motor sequence learning task. We found that the left occipitotemporal cortex (OTC), which previous studies reported to be dysfunctional in dyslexia, can be fractionated into different functional areas: an anterior and lateral area that was activated by both reading and auditory rhyming tasks; a posterior area that was commonly activated by both the reading and the motion perception task and a medial/intermediate area, including the so‐called Visual Word Form Area, which was specifically activated by the reading task. These results show that the left OTC is an area of segregated convergence of different functional systems. We compared our results with the hypoactivation pattern reported for reading in a previous cross‐cultural PET study on 36 dyslexic subjects from three countries. The region of decreased activation in dyslexia overlapped with regions that are specific for reading and those activated during both the auditory rhyming task and the single word and pseudoword reading task described in the present fMRI study. No overlap was found with the activation patterns for the visual motion perception task or for the motor sequence learning task. These observations challenge current theories of dyslexia. Hum Brain Mapp 34:2669–2687, 2013. © 2012 Wiley Periodicals, Inc.     

Modality specific neural correlates of auditory and somatic hallucinations

Somatic hallucinations occur in schizophrenia and other psychotic disorders, although auditory hallucinations are more common. Although the neural correlates of auditory hallucinations have been described in several neuroimaging studies, little is known of the pathophysiology of somatic hallucinations. Functional magnetic resonance imaging (fMRI) was used to compare the distribution of brain activity during somatic and auditory verbal hallucinations, occurring at different times in a 36 year old man with schizophrenia. Somatic hallucinations were associated with activation in the primary somatosensory and posterior parietal cortex, areas that normally mediate tactile perception. Auditory hallucinations were associated with activation in the middle and superior temporal cortex, areas involved in processing external speech. Hallucinations in a given modality seem to involve areas that normally process sensory information in that modality.     

Attention to visual speech gestures enhances hemodynamic activity in the left planum temporale

Observing a speaker's articulatory gestures can contribute considerably to auditory speech perception. At the level of neural events, seen articulatory gestures can modify auditory cortex responses to speech sounds and modulate auditory cortex activity also in the absence of heard speech. However, possible effects of attention on this modulation have remained unclear. To investigate the effect of attention on visual speech-induced auditory cortex activity, we scanned 10 healthy volunteers with functional magnetic resonance imaging (fMRI) at 3 T during simultaneous presentation of visual speech gestures and moving geometrical forms, with the instruction to either focus on or ignore the seen articulations. Secondary auditory cortex areas in the bilateral posterior superior temporal gyrus and planum temporale were active both when the articulatory gestures were ignored and when they were attended to. However, attention to visual speech gestures enhanced activity in the left planum temporale compared to the situation when the subjects saw identical stimuli but engaged in a nonspeech motion discrimination task. These findings suggest that attention to visually perceived speech gestures modulates auditory cortex function and that this modulation takes place at a hierarchically relatively early processing level.     

Spectro-temporal processing during speech perception involves left posterior auditory cortex

This functional magnetic resonance imaging study investigates the neural underpinnings of spectro-temporal integration during speech perception. Participants performed an auditory discrimination task on a set of sine-wave analogues that could be perceived as either nonspeech or speech. Behavioural results revealed a difference in the processing mode; spectro-temporal integration occurred during speech perception, but not when stimuli were perceived as nonspeech. In terms of neuroimaging, we observed an activation increase in the left posterior primary and secondary auditory cortex, namely Heschl's gyrus and planum temporale encroaching onto the superior temporal sulcus, reflecting a shift from auditory to speech perception. This finding demonstrates that the left posterior superior temporal lobe is essential for spectro-temporal processing during speech perception.     

The discrimination of and orienting to speech and non-speech sounds in children with autism

The present study aimed to find out how different stages of cortical auditory processing (sound encoding, discrimination, and orienting) are affected in children with autism. To this end, auditory event-related potentials (ERP) were studied in 15 children with autism and their controls. Their responses were recorded for pitch, duration, and vowel changes in speech stimuli, and for corresponding changes in the non-speech counterparts of the stimuli, while the children watched silent videos and ignored the stimuli. The responses to sound repetition were diminished in amplitude in the children with autism, reflecting impaired sound encoding. The mismatch negativity (MMN), an ERP indexing sound discrimination, was enhanced in the children with autism as far as pitch changes were concerned. This is consistent with earlier studies reporting auditory hypersensitivity and good pitch-processing abilities, as well as with theories proposing enhanced perception of local stimulus features in individuals with autism. The discrimination of duration changes was impaired in these children, however. Finally, involuntary orienting to sound changes, as reflected by the P3a ERP, was more impaired for speech than non-speech sounds in the children with autism, suggesting deficits particularly in social orienting. This has been proposed to be one of the earliest symptoms to emerge, with pervasive effects on later development.