Hearing lips: gamma-band activity during audiovisual speech perception

Auditory pattern changes have been shown to elicit increases in magnetoencephalographic gamma-band activity (GBA) over left inferior frontal cortex, forming part of the putative auditory ventral "what" processing stream. The present study employed a McGurk-type paradigm to assess whether GBA would be associated with subjectively perceived changes even when auditory stimuli remain unchanged. Magnetoencephalograms were recorded in 16 human subjects during audiovisual mismatch perception. Both infrequent visual (auditory /ta/ + visual /pa/) and acoustic deviants (auditory/pa/ + visual /ta/) were compared with frequent audiovisual standards (auditory /ta/ and visual /ta/). Statistical probability mapping revealed spectral amplitude increases at approximately 75 and approximately 78 Hz to visual deviants. GBA to visual deviants peaked 160 ms after auditory stimulus onset over posterior parietal cortex, at 270 ms over occipital areas and at 320 ms over left inferior frontal cortex. The latter GBA enhancement was consistent with the increase observed previously to pure acoustic mismatch, supporting a role of left inferior frontal cortex for the representation of perceived auditory pattern change. The preceding gamma-band changes over posterior areas may reflect processing of incongruent lip movements in visual motion areas and back-projections to earlier visual cortex.     

Cerebral blood flow relationships associated with a difficult tone recognition task in trained normal volunteers

Tone recognition is partially subserved by neural activity in the right frontal and primary auditory cortices. First we determined the brain areas associated with tone perception and recognition. This study then examined how regional cerebral blood flow (rCBF) in these and other brain regions correlates with the behavioral characteristics of a difficult tone recognition task. rCBF changes were assessed using H2(15)O positron emission tomography. Subtraction procedures were used to localize significant change regions and correlational analyses were applied to determine how response times (RT) predicted rCBF patterns. Twelve trained normal volunteers were studied in three conditions: REST, sensory motor control (SMC) and decision (DEC). The SMC-REST contrast revealed bilateral activation of primary auditory cortices, cerebellum and bilateral inferior frontal gyri. DEC-SMC produced significant clusters in the right middle and inferior frontal gyri, insula and claustrum; the anterior cingulate gyrus and supplementary motor area; the left insula/claustrum; and the left cerebellum. Correlational analyses, RT versus rCBF from DEC scans, showed a positive correlation in right inferior and middle frontal cortex; rCBF in bilateral auditory cortices and cerebellum exhibited significant negative correlations with RT These changes suggest that neural activity in the right frontal, superior temporal and cerebellar regions shifts back and forth in magnitude depending on whether tone recognition RT is relatively fast or slow, during a difficult, accurate assessment.      

The neural correlates of conscious vision

Conflicting accounts of the neurobiology of consciousness have emerged from previous imaging studies. Some studies suggest that visual consciousness relates to a distributed network of frontal and partietal regions while others point to localized activity within individual visual areas. While the two positions seem mutually exclusive, timing issues may help reconcile the two. Networks that appear unified in functional magnetic resonance imaging (fMRI) studies may reflect processes that are widely distributed in time. To help resolve this issue, we have investigated timing across a network correlating with consciousness in parallel fMRI and evoked potential (EP) studies of grating stimuli. At threshold, a stimulus is perceived on some occasions but not on others, dissociating sensory input and perception. We have found correlates of consciousness in the occipital lobe at 100 ms and in parietal, frontal, auditory and motor regions from 260 ms onwards. The broad temporal and spatial distribution of activity argues against a unified, distributed fronto-parietal correlate of consciousness. Instead, it suggests that correlates of consciousness are divided into primary and secondary network nodes, with early activity in the occipital lobe correlating with perception and later activity in downstream areas with secondary processes contingent on the outcome of earlier perceptual processing.     

Rapid distributed fronto-parieto-occipital processing stages during working memory in humans

Cortical potentials were recorded from implanted electrodes during a difficult working memory task requiring rapid storage, modification and retrieval of multiple memoranda. Synchronous event-related potentials were generated in distributed occipital, parietal, Rolandic and prefrontal sites beginning approximately 130 ms after stimulus onset and continuing for >500 ms. Coherent phase-locked, event-related oscillations supported interaction between these dorsal stream structures throughout the task period. The Rolandic structures generated early as well as sustained potentials to sensory stimuli in the absence of movement. Activation peaks and phase lags between synaptic populations suggested that perceptual processing occurred exclusively in the visual association cortex from approximately 90 to 130 ms, with its results projected to fronto-parietal areas for interpretation from approximately 130 to 280 ms. The direction of interaction then appeared to reverse from approximately 300 to 400 ms, consistent with mental arithmetic being performed by fronto-parietal areas operating upon a visual scratch pad in the dorsolateral occipital cortex. A second reversal, from approximately 420 to 600 ms, may have represented an updating of memoranda stored in fronto-parietal sites. Lateralized perisylvian oscillations suggested an articulatory loop. Anterior cingulate activity was evoked by feedback signals indicating errors. These results indicate how a fronto-centro-parietal 'central executive' might interact with an occipital visual scratch pad, perisylvian articulatory loop and limbic monitor to implement the sequential stages of a complex mental operation.     

Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study

Multisensory object-recognition processes were investigated by examining the combined influence of visual and auditory inputs upon object identification--in this case, pictures and vocalizations of animals. Behaviorally, subjects were significantly faster and more accurate at identifying targets when the picture and vocalization were matched (i.e. from the same animal), than when the target was represented in only one sensory modality. This behavioral enhancement was accompanied by a modulation of the evoked potential in the latency range and general topographic region of the visual evoked N1 component, which is associated with early feature processing in the ventral visual stream. High-density topographic mapping and dipole modeling of this multisensory effect were consistent with generators in lateral occipito-temporal cortices, suggesting that auditory inputs were modulating processing in regions of the lateral occipital cortices. Both the timing and scalp topography of this modulation suggests that there are multisensory effects during what is considered to be a relatively early stage of visual object-recognition processes, and that this modulation occurs in regions of the visual system that have traditionally been held to be unisensory processing areas. Multisensory inputs also modulated the visual 'selection-negativity', an attention dependent component of the evoked potential this is usually evoked when subjects selectively attend to a particular feature of a visual stimulus.     

Beyond superior temporal cortex: intersubject correlations in narrative speech comprehension

The role of superior temporal cortex in speech comprehension is well established, but the complete network of regions involved in understanding language in ecologically valid contexts is less clearly understood. In a functional magnetic resonance imaging (fMRI) study, we presented 24 subjects with auditory or audiovisual narratives, and used model-free intersubject correlational analyses to reveal brain areas that were modulated in a consistent way across subjects during the narratives. Conventional comparisons to a resting state were also performed. Both analyses showed the expected recruitment of superior temporal areas, however, the intersubject correlational analyses also revealed an extended network of areas involved in narrative speech comprehension. Two findings stand out in particular. Firstly, many areas in the "default mode" network (typically deactivated relative to rest) were systematically modulated by the time-varying properties of the auditory or audiovisual input. These areas included the anterior cingulate and adjacent medial frontal cortex, and the posterior cingulate and adjacent precuneus. Secondly, extensive bilateral inferior frontal and premotor regions were implicated in auditory as well as audiovisual language comprehension. This extended network of regions may be important for higher-level linguistic processes, and interfaces with extralinguistic cognitive, affective, and interpersonal systems.     

Implicitly perceived vocal attractiveness modulates prefrontal cortex activity

Social interactions involve more than "just" language. As important is a more primitive nonlinguistic mode of communication acting in parallel with linguistic processes and driving our decisions to a much higher degree than is generally suspected. Amongst the "honest signals" that influence our behavior is perceived vocal attractiveness. Not only does vocal attractiveness reflect important biological characteristics of the speaker, it also influences our social perceptions according to the "what sounds beautiful is good" phenomenon. Despite the widespread influence of vocal attractiveness on social interactions revealed by behavioral studies, its neural underpinnings are yet unknown. We measured brain activity while participants listened to a series of vocal sounds ("ah") and performed an unrelated task. We found that voice-sensitive auditory and inferior frontal regions were strongly correlated with implicitly perceived vocal attractiveness. While the involvement of auditory areas reflected the processing of acoustic contributors to vocal attractiveness ("distance to mean" and spectrotemporal regularity), activity in inferior prefrontal regions (traditionally involved in speech processes) reflected the overall perceived attractiveness of the voices despite their lack of linguistic content. These results suggest the strong influence of hidden nonlinguistic aspects of communication signals on cerebral activity and provide an objective measure of this influence.     

Mental visual synthesis is originated in the fronto-temporal network of the left hemisphere

Mental visual synthesis is the capacity for experiencing, constructing, or manipulating 'mental imagery'. To investigate brain networks involved in mental visual synthesis, brain activity was measured in right-handed healthy volunteers during mental imagery tasks, in which the subjects were instructed to imagine a novel object, that does not exist in the real world, by composing it from two visually presented words associated with a real object or two achromatic line drawings of a real object, using functional magnetic resonance imaging (fMRI). Both tasks activated the same areas in the inferior frontal and inferior temporal cortices of the left hemisphere. Our results indicate that the source of mental visual synthesis may be formed by activity of a brain network consisting of these areas, which are also involved in semantic operations and visual imagery.     

Brain mechanisms mediating auditory attentional capture in humans

The ability to detect and preferentially process salient auditory stimuli, even when irrelevant to a current task, is often critical for adaptive behavior. This stimulus-driven allocation of processing resources is known as "attentional capture." Here we used functional magnetic resonance imaging in humans to investigate brain activity and behavioral effects related to such auditory attentional capture. Participants searched a sequence of tones for a target tone that was shorter or longer than the nontarget tones. An irrelevant singleton feature in the tone sequence resulted in behavioral interference (attentional capture) and activation of parietal and prefrontal cortices only when the singleton was associated with a nontarget tone (nontarget singleton) and not when associated with a target tone (target singleton). In contrast, the presence (vs. absence) of a singleton feature in the sequence was associated with activation of frontal and temporal loci previously associated with auditory change detection. These results suggest that a ventral network involving superior temporal and inferior frontal cortices responds to acoustic variability, regardless of attentional significance, but a dorsal frontoparietal network responds only when a feature singleton captures attention.     

Hearing lips and seeing voices: how cortical areas supporting speech production mediate audiovisual speech perception

Observing a speaker's mouth profoundly influences speech perception. For example, listeners perceive an "illusory" "ta" when the video of a face producing /ka/ is dubbed onto an audio /pa/. Here, we show how cortical areas supporting speech production mediate this illusory percept and audiovisual (AV) speech perception more generally. Specifically, cortical activity during AV speech perception occurs in many of the same areas that are active during speech production. We find that different perceptions of the same syllable and the perception of different syllables are associated with different distributions of activity in frontal motor areas involved in speech production. Activity patterns in these frontal motor areas resulting from the illusory "ta" percept are more similar to the activity patterns evoked by AV(/ta/) than they are to patterns evoked by AV(/pa/) or AV(/ka/). In contrast to the activity in frontal motor areas, stimulus-evoked activity for the illusory "ta" in auditory and somatosensory areas and visual areas initially resembles activity evoked by AV(/pa/) and AV(/ka/), respectively. Ultimately, though, activity in these regions comes to resemble activity evoked by AV(/ta/). Together, these results suggest that AV speech elicits in the listener a motor plan for the production of the phoneme that the speaker might have been attempting to produce, and that feedback in the form of efference copy from the motor system ultimately influences the phonetic interpretation.     

Changes in auditory cortex parallel rapid perceptual learning

Learning perceptual skills is characterized by rapid improvements in performance within the first hour of training (fast perceptual learning) followed by more gradual improvements that take place over several daily practice sessions (slow perceptual learning). Although it is widely accepted that slow perceptual learning is accompanied by enhanced stimulus representation in sensory cortices, there is considerable controversy about the neural substrates underlying early and rapid improvements in learning perceptual skills. Here we measured event-related brain potentials while listeners were presented with 2 phonetically different vowels. Listeners' ability to identify both vowels improved gradually during the first hour of testing and was paralleled by enhancements in an early evoked response ( approximately 130 ms) localized in the right auditory cortex and a late evoked response ( approximately 340 ms) localized in the right anterior superior temporal gyrus and/or inferior prefrontal cortex. These neuroplastic changes depended on listeners' attention and were preserved only if practice was continued; familiarity with the task structure (procedural learning) was not sufficient. We propose that the early increases in cortical responsiveness reflect goal-directed changes in the tuning properties of auditory neurons involved in parsing concurrent speech signals. Importantly, the neuroplastic changes occurred rapidly, demonstrating the flexibility of human speech segregation mechanisms.     

Sequential effects of increasing propofol sedation on frontal and temporal cortices as indexed by auditory event-related potentials

BACKGROUND: It is an open question whether cognitive processes of auditory perception that are mediated by functionally different cortices exhibit the same sensitivity to sedation. The auditory event-related potentials P1, mismatch negativity (MMN), and early right anterior negativity (ERAN) originate from different cortical areas and reflect different stages of auditory processing. The P1 originates mainly from the primary auditory cortex. The MMN is generated in or in the close vicinity of the primary auditory cortex but is also dependent on frontal sources. The ERAN mainly originates from frontal generators. The purpose of the study was to investigate the effects of increasing propofol sedation on different stages of auditory processing as reflected in P1, MMN, and ERAN. METHODS: The P1, the MMN, and the ERAN were recorded preoperatively in 18 patients during four levels of anesthesia adjusted with target-controlled infusion: awake state (target concentration of propofol 0.0 microg/ml), light sedation (0.5 microg/ml), deep sedation (1.5 microg/ml), and unconsciousness (2.5-3.0 microg/ml). Simultaneously, propofol anesthesia was assessed using the Bispectral Index. RESULTS: Propofol sedation resulted in a progressive decrease in amplitudes and an increase of latencies with a similar pattern for MMN and ERAN. MMN and ERAN were elicited during sedation but were abolished during unconsciousness. In contrast, the amplitude of the P1 was unchanged by sedation but markedly decreased during unconsciousness. CONCLUSION: The results indicate differential effects of propofol sedation on cognitive functions that involve mainly the auditory cortices and cognitive functions that involve the frontal cortices.     

Spontaneous activity associated with primary visual cortex: a resting-state FMRI study

Brain functions during the resting state have attracted considerable attention in the past several years. However, little has been known about spontaneous activity in the sensory cortices in the task-free state. This study used functional magnetic resonance imaging (fMRI) to investigate the existence of spontaneous activity in the primary visual areas (PVA) of normal-sighted subjects and to explore the physiological implications of such activity. Our results revealed that we were able to detect spontaneous activity, which was nonrandom in that it was distinctly clustered both temporally and spatially in the PVA of each subject. In addition, the neural network associated with the PVA-related spontaneous activity included the visual association areas, the precuneus, the precentral/postcentral gyrus, the middle frontal gyrus, the fusiform gyrus, the inferior/middle temporal gyrus, and the parahippocampal gyrus. After considering the functions of these regions, we speculated that the PVA-related spontaneous activity may be associated with memory-related mental imagery and/or visual memory consolidation processes. These findings confirm the presence of spontaneous activity in the PVA and related brain areas. This confirmation supports the perspective that brain is a system intrinsically operating on its own, and sensory information interacts with rather than determines the operation of the system.     

Similar frontal and distinct posterior cortical regions mediate visual and auditory perceptual awareness

Activity in ventral visual cortex is a consistent neural correlate of visual consciousness. However, activity in this area seems insufficient to produce awareness without additional involvement of frontoparietal regions. To test the generality of the frontoparietal response, neural correlates of auditory awareness were investigated in a paradigm that previously has revealed frontoparietal activity during conscious visual perception. A within-experiment comparison showed that frontal regions were related to both visual and auditory awareness, whereas parietal activity was correlated with visual awareness and superior temporal activity with auditory awareness. These results indicate that frontal regions interact with specific posterior regions to produce awareness in different sensory modalities.     

Retinotopy and attention in human occipital, temporal, parietal, and frontal cortex

Novel mapping stimuli composed of biological motion figures were used to study the extent and layout of multiple retinotopic regions in the entire human brain and to examine the independent manipulation of retinotopic responses by visual stimuli and by attention. A number of areas exhibited retinotopic activations, including full or partial visual field representations in occipital cortex, the precuneus, motion-sensitive temporal cortex (extending into the superior temporal sulcus), the intraparietal sulcus, and the vicinity of the frontal eye fields in frontal cortex. Early visual areas showed mainly stimulus-driven retinotopy; parietal and frontal areas were driven primarily by attention; and lateral temporal regions could be driven by both. We found clear spatial specificity of attentional modulation not just in early visual areas but also in classical attentional control areas in parietal and frontal cortex. Indeed, strong spatiotopic activity in these areas could be evoked by directed attention alone. Conversely, motion-sensitive temporal regions, while exhibiting attentional modulation, also responded significantly when attention was directed away from the retinotopic stimuli.     

Reading salt activates gustatory brain regions: FMRI evidence for semantic grounding in a novel sensory modality

Because many words are typically used in the context of their referent objects and actions, distributed cortical circuits for these words may bind information about their form with perceptual and motor aspects of their meaning. Previous work has demonstrated such semantic grounding for sensorimotor, visual, auditory, and olfactory knowledge linked to words, which is manifest in activation of the corresponding areas of the cortex. Here, we explore the brain basis of gustatory semantic links of words whose meaning is primarily related to taste. In a blocked functional magnetic resonance imaging design, Spanish taste words and control words matched for a range of factors (including valence, arousal, imageability, frequency of use, number of letters and syllables) were presented to 59 right-handed participants in a passive reading task. Whereas all the words activated the left inferior frontal (BA44/45) and the posterior middle and superior temporal gyri (BA21/22), taste-related words produced a significantly stronger activation in these same areas and also in the anterior insula, frontal operculum, lateral orbitofrontal gyrus, and thalamus among others. As these areas comprise primary and secondary gustatory cortices, we conclude that the meaning of taste words is grounded in distributed cortical circuits reaching into areas that process taste sensations.     

New Evidence for Distinct Right and Left Brain Systems for Deductive versus Probabilistic Reasoning.

Deductive and probabilistic reasoning are central to cognition but the functional neuroanatomy underlying them is poorly understood. The present study contrasted these two kinds of reasoning via positron emission tomography. Relying on changes in instruction and psychological 'set', deductive versus probabilistic reasoning was induced using identical stimuli. The stimuli were arguments in propositional calculus not readily solved via mental diagrams. Probabilistic reasoning activated mostly left brain areas whereas deductive activated mostly right. Deduction activated areas near right brain homologues of left language areas in middle temporal lobe, inferior frontal cortex and basal ganglia, as well as right amygdala, but not spatial-visual areas. Right hemisphere activations in the deduction task cannot be explained by spill-over from overtaxed, left language areas. Probabilistic reasoning was mostly associated with left hemispheric areas in inferior frontal, posterior cingulate, parahippocampal, medial temporal, and superior and medial prefrontal cortices. The foregoing regions are implicated in recalling and evaluating a range of world knowledge, operations required during probabilistic thought. The findings confirm that deduction and induction are distinct processes, consistent with psychological theories enforcing their partial separation. The results also suggest that, except for statement decoding, deduction is largely independent of language, and that some forms of logical thinking are non-diagrammatic.     

Learning-induced changes in the cerebral processing of voice identity

Temporal voice areas showing a larger activity for vocal than non-vocal sounds have been identified along the superior temporal sulcus (STS); more voice-sensitive areas have been described in frontal and parietal lobes. Yet, the role of voice-sensitive regions in representing voice identity remains unclear. Using a functional magnetic resonance adaptation design, we aimed at disentangling acoustic- from identity-based representations of voices. Sixteen participants were scanned while listening to pairs of voices drawn from morphed continua between 2 initially unfamiliar voices, before and after a voice learning phase. In a given pair, the first and second stimuli could be identical or acoustically different and, at the second session, perceptually similar or different. At both sessions, right mid-STS/superior temporal gyrus (STG) and superior temporal pole (sTP) showed sensitivity to acoustical changes. Critically, voice learning induced changes in the acoustical processing of voices in inferior frontal cortices (IFCs). At the second session only, right IFC and left cingulate gyrus showed sensitivity to changes in perceived identity. The processing of voice identity appears to be subserved by a large network of brain areas ranging from the sTP, involved in an acoustic-based representation of unfamiliar voices, to areas along the convexity of the IFC for identity-related processing of familiar voices.     

Convergence of neural systems processing stimulus associations and coordinating motor responses

A sensory-sensory learning paradigm was used to measure neural changes in humans during acquisition of an association between an auditory and visual stimulus. Three multivariate partial least-squares (PLS) analyses of positron emission tomography data identified distributed neural systems related to (i) processing the significance of the auditory stimulus, (ii) mediating the acquisition of the behavioral response, and (iii) the spatial overlap between these two systems. The system that processed the significance of the tone engaged primarily right hemisphere regions and included dorsolateral prefrontal cortex, putamen, and inferior parietal and temporal cortices. Activity changes in left occipital cortex were also identified, most likely reflecting the learned expectancy of the upcoming visual event. The system related to behavior was similar to that which coded the significance of the tone, including dorsal occipital cortex. The PLS analysis of the concordance between these two systems showed substantial regional overlap, and included occipital, dorsolateral prefrontal, and limbic cortices. However, activity in dorsomedial prefrontal cortex was strictly related to processing the auditory stimulus and not to behavior. Taken together, the PLS analyses identified a system that contained a sensory-motor component (comprised of occipital, temporal association and sensorimotor cortices) and a medial prefrontallimbic component, that as a group simultaneously embodied the learning-related response to the stimuli and the subsequent change in behavior.      

When that tune runs through your head: a PET investigation of auditory imagery for familiar melodies.

The present study used positron emission tomography (PET) to examine the cerebral activity pattern associated with auditory imagery for familiar tunes. Subjects either imagined the continuation of nonverbal tunes cued by their first few notes, listened to a short sequence of notes as a control task, or listened and then reimagined that short sequence. Subtraction of the activation in the control task from that in the real-tune imagery task revealed primarily right-sided activation in frontal and superior temporal regions, plus supplementary motor area (SMA). Isolating retrieval of the real tunes by subtracting activation in the reimagine task from that in the real-tune imagery task revealed activation primarily in right frontal areas and right superior temporal gyrus. Subtraction of activation in the control condition from that in the reimagine condition, intended to capture imagery of unfamiliar sequences, revealed activation in SMA, plus some left frontal regions. We conclude that areas of right auditory association cortex, together with right and left frontal cortices, are implicated in imagery for familiar tunes, in accord with previous behavioral, lesion and PET data. Retrieval from musical semantic memory is mediated by structures in the right frontal lobe, in contrast to results from previous studies implicating left frontal areas for all semantic retrieval. The SMA seems to be involved specifically in image generation, implicating a motor code in this process.