The human temporal lobe integrates facial form and motion: evidence from fMRI and ERP studies

Physiological studies in humans and monkeys indicate that the posterior temporal cortex is active when viewing the movements of others. Here we tested the premise that this region integrates form and motion information by presenting both natural and line-drawn displays of moving faces and motion controls where motion was continuously presented in the same part of the visual field. The cortex in and near the STS and on the fusiform gyrus (FG) responded to both types of face stimuli, but not to the controls, in a functional magnetic resonance imaging study in 10 normal subjects. The response in the STS to both types of facial motion was equal in magnitude, whereas in the FG the natural image of the face produced a significantly greater response than that of the line-drawn face. In a subsequent recording session, the electrical activity of the brain was recorded in the same subjects to the same activation task. Significantly larger event-related potentials (ERPs) to both types of moving faces were observed over the posterior temporal scalp compared to the motion controls at around 200 ms postmotion onset. Taken together, these data suggest that regions of temporal cortex actively integrate form and motion information-a process largely independent of low-level visual processes such as changes in local luminance and contrast.     

Processing social aspects of human gaze: a combined fMRI-DTI study

Human gaze is a critical social cue that can reveal intentions and dispositions of others. The right posterior superior temporal sulcus (pSTS) is thought to be critically involved in processing eye gaze information. We combined diffusion tensor imaging (DTI) and functional magnetic resonance imaging (fMRI) to identify direct neural connections of right pSTS and to determine how these connections are modulated by the social significance of perceived gaze shifts. Participants saw faces with direct or averted gaze during event-related fMRI. Half of these faces remained static, and half displayed a dynamic gaze shift either towards or away from the subject. Social attention (dynamic gaze shifts towards the observer) not only increased activity in right pSTS, but also its functional connectivity with the right anterior insula (aIns) and right fusiform gyrus (FG). However, direct fiber connections from pSTS were demonstrated by DTI for the right aIns, but not the right FG. Moreover, the right FG responded to eye motion irrespective of direction and social significance; whereas the right aIns was selectively sensitive to social significance (i.e. gaze shifts towards the observer), but not generally to eye motion. We conclude that the social aspects of mutual gaze contact are processed by direct fiber pathways between right pSTS and right aIns; whereas increased connectivity with FG could reflect an enhanced perceptual analysis of changing facial features in dynamic gaze conditions and involves indirect fiber pathways with pSTS, perhaps via motion-selective regions in middle temporal (MT) gyrus that exhibited strong white-matter connections with both pSTS and FG and could thus provide inputs to these two areas.     

Dynamics of processing invisible faces in the brain: automatic neural encoding of facial expression information

The fusiform face area (FFA) and the superior temporal sulcus (STS) are suggested to process facial identity and facial expression information respectively. We recently demonstrated a functional dissociation between the FFA and the STS as well as correlated sensitivity of the STS and the amygdala to facial expressions using an interocular suppression paradigm [Jiang, Y., He, S., 2006. Cortical responses to invisible faces: dissociating subsystems for facial-information processing. Curr. Biol. 16, 2023-2029.]. In the current event-related brain potential (ERP) study, we investigated the temporal dynamics of facial information processing. Observers viewed neutral, fearful, and scrambled face stimuli, either visibly or rendered invisible through interocular suppression. Relative to scrambled face stimuli, intact visible faces elicited larger positive P1 (110-130 ms) and larger negative N1 or N170 (160-180 ms) potentials at posterior occipital and bilateral occipito-temporal regions respectively, with the N170 amplitude significantly greater for fearful than neutral faces. Invisible intact faces generated a stronger signal than scrambled faces at 140-200 ms over posterior occipital areas whereas invisible fearful faces (compared to neutral and scrambled faces) elicited a significantly larger negative deflection starting at 220 ms along the STS. These results provide further evidence for cortical processing of facial information without awareness and elucidate the temporal sequence of automatic facial expression information extraction.     

Seen Gaze-Direction Modulates Fusiform Activity and Its Coupling with Other Brain Areas during Face Processing

Gaze-contact is often a preliminary to social interaction and so constitutes a signal for the allocation of processing resources to the gazing face. We investigated how gaze direction influences face processing in an fMRI study, where seen gaze and head direction could independently be direct or deviated. Direct relative to averted gaze elicited stronger activation for faces in ventral occipitotemporal cortices around the fusiform gyrus, regardless of head orientation. Moreover, direct gaze led to greater correlation between activity in the fusiform and the amygdala, a region associated with emotional responses and stimulus saliency. By contrast, faces with averted gaze (again, regardless of head orientation) yielded increased correlation between activity in the fusiform and the intraparietal sulcus, a region associated with shifting attention to the periphery.     

Developmental differences in the neural bases of the face inversion effect show progressive tuning of face-selective regions to the upright orientation

Face inversion hinders face processing in adults, while not affecting children in the same way. This fMRI study examines the neural underpinnings of the behavioral face inversion effect (FIE) from childhood to adulthood, and how face-selective regions in the brain may change with development. Adults, children, and teens performed a facial expression decision on upright and inverted face stimuli. In the right hemisphere (RH) all age groups showed similar profiles of neural activation for upright faces, but important developmental differences occured for inverted faces. For inverted faces, adults, and to a lesser degree teens, exhibited decreased levels of activity in the face-selective, right lateral fusiform gyrus (LFG). However, children exhibited greater activation for inverted than for upright faces in the same region. We found similar, but less robust, developmental trends in the right superior temporal sulcus (STS) and medial fusiform gyrus (MFG). Furthermore, the present study identifies the right LFG as the primary neural correlate of the behavioral FIE, and therefore of face processing expertise, by showing a significant correlation between the behavioral FIE and the neural FIE only in this region. Finally, the present findings shed some light on at least one of the possible mechanisms underlying the development of face processing expertise, by suggesting a progressive tuning of face-selective regions in the right hemisphere to the upright orientation, that extends well into adolescence.     

Increased amygdala activation to averted versus direct gaze in humans is independent of valence of facial expression

A recent study in monkeys showed that averted gaze displayed by conspecifics activates the amygdala regardless of the valence of facial expression (angry, neutral, appeasing). In contrast to this result, previous findings on amygdala activation to gaze information in humans have been mainly interpreted in terms of threat-related processing of facial signals. In the present event-related functional magnetic resonance imaging study with humans, we investigated brain responses to angry, happy, and neutral faces with either direct or averted gaze. Averted versus direct gaze induced increased amygdala responses regardless of facial expression. This finding suggests a critical role of the amygdala in gaze processing independently of the valence of facial expression. Furthermore, the effect of gaze on amygdalar responses seems to be preserved across nonhuman and human primates.     

Brain responses to dynamic facial expressions of pain

The facial expression of pain is a prominent non-verbal pain behaviour, unique and distinct from the expression of basic emotions. Yet, little is known about the neurobiological basis for the communication of pain. Here, subjects performed a sex-discrimination task while we investigated neural responses to implicit processing of dynamic visual stimuli of male or female faces displaying pain or angry expressions, matched on expression intensity and compared to neutral expression. Stimuli were presented in a mixed blocked/event-related design while blood oxygenation level dependent (BOLD) signal was acquired using whole-brain functional magnetic resonance imaging (fMRI) at 1.5 Tesla. Comparable sustained responses to pain and angry faces were found in the superior temporal sulcus (STS). Stronger transient activation was also observed to male expression of pain (Vs neutral and anger) in high-order visual areas (STS and fusiform face area) and in emotion-related areas including the amygdala (highest peak t-value=10.8), perigenual anterior cingulate cortex (ACC), and SI. Male pain compared to anger expression also activated the ventromedial prefrontal cortex, SII/posterior insula and anterior insula. This is consistent with the hypothesis that the implicit processing of male pain expression triggers an emotional reaction characterized by a threat-related response. Unexpectedly, several areas responsive to male expression, including the amygdala, perigenual ACC, and somatosensory areas, showed a decrease in activation to female pain faces (Vs neutral). This sharp contrast in the response to male and female faces suggests potential differences in the socio-functional role of pain expression in males and females.     

Is voice processing species-specific in human auditory cortex? An fMRI study

Recent studies suggested a sensitivity of regions of the human superior temporal sulcus (STS) to the sound of the human voice. However, the question of the species specificity of this response is still open. Healthy adult volunteers were scanned in an event-related fMRI design to compare responses in the STS to human and animal vocalizations, as well as to control nonvocal sounds (e.g., musical instruments). Bilateral activation of anterior STS was observed for human vocalizations, when contrasted with both nonvocal sounds and animal vocalizations. Animal vocalizations, compared to nonvocal sounds, elicited a more restricted left STS activation, although this region responded even more strongly to human vocalizations. This study provides the first evidence suggesting a species specificity in STS responses to vocalizations in humans.     

Scale invariant adaptation in fusiform face-responsive regions

Several functional neuroimaging studies have observed response adaptation in face-sensitive regions when repeating identical face stimuli. To address whether this was due to low-level stimulus properties or facial identity, we decomposed pictures of faces into pictures preserving only the lower or higher parts of the normal frequency spectrum. In an event-related functional neuroimaging study, pairs of such pictures were sequentially presented that showed the same or different persons in the same or different frequency bands. This factorial design allowed to separate effects related to repetition of personal identity from those related to identical stimulus properties. In a random effects group analysis, activation in the right fusiform region was affected by repetition of personal identity regardless of changing or constant spatial scale. Responses in the more medial and posterior fusiform and lingual regions adapted with repetition of the same frequency band. An analysis in regions of interest determined individually as face responsive showed that repetition decreases for the same faces in fusiform face-responsive regions generalized across spatial frequency bands. Our results therefore point to a role of this area in discriminating individual faces at a level of representation that is invariant to changes in low-level stimulus properties, as spatial scale. The same invariance could not be detected in more posterior occipital face-responsive regions.     

The chronoarchitecture of human sexual desire: a high-density electrical mapping study

Recent neuroimaging research suggests that human sexual desire (SD) recruits both the limbic system and higher-order cognitive brain areas. Because of the temporal limitation of this technique, the chronoarchitecture of SD remains however unresolved. Here, we investigated the spatio-temporal dynamics of SD by combining a behavioral desire decision task with high-density visual event-related potential (VEP) recordings and brain source estimations. VEPs were recorded from thirteen healthy participants when presented with pictures from two different stimulus categories (i.e., high and low desirability). In agreement with the literature, behavioral results showed that participants were faster to rate non-desired stimuli than desired stimuli (p=0.028). Electrophysiological results extended these behavioral data. Group-averaged VEPs peaked at 90 to 140 ms (P100), at 142 to 220 ms (N200), and at 222 to 360 ms (P300). Desired stimuli (DS) were distinguished from non-desired stimuli (NDS) over the N200 period, notably from 142 to 187 ms. Over this time period, DS processing was characterized by a significant scalp potential field. Although both conditions (DS and NDS) showed the recruitment of the occipito-temporal region (including the extrastriate body area, EBA), LAURA source estimation of the DS scalp potential field revealed a more right-lateralized current source density maximum in the posterior superior temporal sulcus (STS) extending to the temporo-parietal junction (TPJ). The recruitment of STS and TPJ for desired stimuli indicates that these brain areas, known to be respectively involved in social cognition, attention, integration of body-related information and self-processing, play a crucial role for the coding of desirability of visual sexual human stimuli within the first 200 ms after stimulus onset. These findings support the hypothesis that complex cognitive processing for desire occurs much faster than previously thought and open new perspectives with respect to the role of both bottom-up and top-down mechanisms in visual processing of sexual stimuli.     

Taking an "intentional stance" on eye-gaze shifts: a functional neuroimaging study of social perception in children

During middle childhood, children develop an increasing understanding of intentions and other social information conveyed through dynamic facial cues such as changes in eye-gaze direction. Recent work in our laboratory has focused on using functional magnetic resonance imaging (fMRI) in adults to map the neural circuitry subserving the visual analysis of others' actions and the intentions underlying these actions. In these studies, the superior temporal sulcus (STS) region has been continually implicated in processing shifts in eye gaze. Further, these studies have indicated that STS activity is modulated by the context within which eye-gaze shifts occur, suggesting that this region is involved in social perception via its role in the analysis of the intentions of observed actions. Still, no studies have investigated the neural circuitry supporting eye-gaze processing in children. We used event-related fMRI to examine brain activity in 7- to 10-year-old healthy children observing an animated virtual actor who shifted her eyes towards either a target object or empty space. Consistent with prior studies in adults, the STS, middle temporal gyrus, and inferior parietal lobule were sensitive to the intentions underlying the stimulus character's eye movements. These findings suggest that the neural circuitry underlying the processing of eye gaze and the detection of intentions conveyed through shifts in eye gaze in children are similar to that found previously in adults. We discuss these findings and potential implications for mapping the neurodevelopment of the social cognition and social perception abnormalities characteristic of autism.     

The neural basis of social tactics: An fMRI study

One of the most powerful ways of succeeding in complex social interactions is to read the minds of companions and stay a step ahead of them. In order to assess neural responses to reciprocal mind reading in socially strained human relationships, we used a 3-T scanner to perform an event-related functional magnetic resonance imaging study in 16 healthy subjects who participated in the game of Chicken. Statistical parametric mapping showed that the counterpart effect (human minus computer) exclusively activated the medial frontal area corresponding to the anterior paracingulate cortex (PCC) and the supramarginal gyrus neighboring the posterior superior temporal sulcus (STS). Furthermore, when we analyzed the data to evaluate whether the subjects made risky/aggressive or safe/reconciliatory choices, the posterior STS showed that the counterpart had a reliable effect regardless of risky or safe decisions. In contrast, a significant opponent x selection interaction was revealed in the anterior PCC. Based on our findings, it could be inferred that the posterior STS and the anterior PCC play differential roles in mentalizing; the former serves as a general mechanism for mentalizing, while the latter is exclusively involved in socially risky decisions.     

Controlling for individual differences in fMRI brain activation to tones, syllables, and words

Previous neuroimaging studies have consistently reported bilateral activation to speech stimuli in the superior temporal gyrus (STG) and have identified an anteroventral stream of speech processing along the superior temporal sulcus (STS). However, little attention has been devoted to the possible confound of individual differences in hemispheric dominance for speech. The present study was designed to test for speech-selective activation while controlling for inter-individual variance in auditory laterality, by using only subjects with at least 10% right ear advantage (REA) on the dichotic listening test. Eighteen right-handed, healthy male volunteers (median age 26) participated in the study. The stimuli were words, syllables, and sine wave tones (220-2600 Hz), presented in a block design. Comparing words > tones and syllables > tones yielded activation in the left posterior MTG and the lateral STG (upper bank of STS). In the right temporal lobe, the activation was located in the MTG/STS (lower bank). Comparing left and right temporal lobe cluster sizes from the words > tones and syllables > tones contrasts on single-subject level demonstrated a statistically significant left lateralization for speech sound processing in the STS/MTG area. The asymmetry analyses suggest that dichotic listening may be a suitable method for selecting a homogenous group of subjects with respect to left hemisphere language dominance.     

Sex differences in the neural correlates of child facial resemblance: an event-related fMRI study

Detection of genetic relatedness (i.e. kinship) impacts the social, parental, and sexual behavior of many species. In humans, self-referent phenotype matching based on facial resemblance may indicate kinship. For example, faces that resemble ours are perceived as more trustworthy and attractive. Sex differences in behavioral reactions to facial resemblance among children have also been demonstrated and are consistent with evolutionary theory suggesting that resemblance might serve as a paternity cue. Using event-related fMRI, we show that specific regions of the brain are implicated in processing facial resemblance and a sex difference in cortical response to facial resemblance expressed in children. We found a consistent activation in the fusiform gyrus across all face conditions, which is consistent with previous research on face processing. There were no sex differences in overall response to faces in the fusiform gyrus, and also to faces that did not resemble subjects. When resemblance was not modeled, females showed greater activation to child faces than males. Consistent with parental investment theory and theories of sexual selection, males showed greater cortical activity than females in response to children's faces that resembled them. These data suggest natural selection may have crafted a sexually differentiated neuro-sensory module implicated in detection of facial resemblance that may serve as a kin detection and paternity cue. This process may capitalize on neural substrates involved in self-referent processing and familiarity detection.     

The neurophysiology of human biological motion processing: a high-density electrical mapping study

The neural processing of biological motion (BM) is of profound experimental interest since it is often through the movement of another that we interpret their immediate intentions. Neuroimaging points to a specialized cortical network for processing biological motion. Here, high-density electrical mapping and source-analysis techniques were employed to interrogate the timing of information processing across this network. Participants viewed point-light-displays depicting standard body movements (e.g. jumping), while event-related potentials (ERPs) were recorded and compared to ERPs to scrambled motion control stimuli. In a pair of experiments, three major phases of BM-specific processing were identified: 1) The earliest phase of BM-sensitive modulation was characterized by a positive shift of the ERP between 100 and 200 ms after stimulus onset. This modulation was observed exclusively over the right hemisphere and source-analysis suggested a likely generator in close proximity to regions associated with general motion processing (KO/hMT). 2) The second phase of BM-sensitivity occurred from 200 to 350 ms, characterized by a robust negative-going ERP modulation over posterior middle temporal regions bilaterally. Source-analysis pointed to bilateral generators at or near the posterior superior temporal sulcus (STS). 3) A third phase of processing was evident only in our second experiment, where participants actively attended the BM aspect of the stimuli, and was manifest as a centro-parietal positive ERP deflection, likely related to later cognitive processes. These results point to very early sensory registration of biological motion, and highlight the interactive role of the posterior STS in analyzing the movements of other living organisms.     

The functional anatomy of inspection time: an event-related fMRI study

Twenty healthy young adults underwent functional magnetic resonance imaging (fMRI) of the brain while performing a visual inspection time task. Inspection time is a forced-choice, two-alternative visual backward-masking task in which the subject is briefly shown two parallel vertical lines of markedly different lengths and must decide which is longer. As stimulus duration decreases, performance declines to chance levels. Individual differences in inspection time correlate with higher cognitive functions. An event-related design was used. The hemodynamic (blood oxygenation level-dependent; BOLD) response was computed as both a function of the eight levels of stimulus duration, from 6 ms (where performance is almost at chance) to 150 ms (where performance is nearly perfect), and a function of the behavioral responses. Random effects analysis showed that the difficulty of the visual discrimination was related to bilateral activation in the inferior fronto-opercular cortex, superior/medial frontal gyrus, and anterior cingulate gyrus, and bilateral deactivation in the posterior cingulate gyrus and precuneus. Examination of the time courses of BOLD responses showed that activation was related specifically to the more difficult, briefer stimuli and that deactivation was found across most stimulus levels. Functional connectivity suggested the existence of two networks. One comprised the fronto-opercular area, intrasylvian area, medial frontal gyrus, and the anterior cingulate cortex (ACC), possibly associated with processing of visually degraded percepts. A posterior network of sensory-related and associative regions might subserve processing of a visual discrimination task that has high processing demands and combines several fundamental cognitive domains. fMRI can thus reveal information about the neural correlates of mental events which occur over very short durations.     

The neural mechanism associated with the processing of onomatopoeic sounds

This event-related fMRI study was conducted to examine the blood-oxygen-level-dependent responses to the processing of auditory onomatopoeic sounds. We used a sound categorization task in which the participants heard four types of stimuli: onomatopoeic sounds, nouns (verbal), animal (nonverbal) sounds, and pure tone/noise (control). By discriminating between the categories of target sounds (birds/nonbirds), the nouns resulted in activations in the left anterior superior temporal gyrus (STG), whereas the animal sounds resulted in activations in the bilateral superior temporal sulcus (STS) and the left inferior frontal gyrus (IFG). In contrast, the onomatopoeias activated extensive brain regions, including the left anterior STG, the region from the bilateral STS to the middle temporal gyrus, and the bilateral IFG. The onomatopoeic sounds showed greater activation in the right middle STS than did the nouns and environmental sounds. These results indicate that onomatopoeic sounds are processed by extensive brain regions involved in the processing of both verbal and nonverbal sounds. Thus, we can posit that onomatopoeic sounds can serve as a bridge between nouns and animal sounds. This is the first evidence to demonstrate the way in which onomatopoeic sounds are processed in the human brain.     

Differential selectivity for dynamic versus static information in face-selective cortical regions

Neuroimaging studies have identified multiple face-selective regions in human cortex but the functional division of labor between these regions is not yet clear. A central hypothesis, with some empirical support, is that face-selective regions in the superior temporal sulcus (STS) are particularly responsive to dynamic information in faces, whereas the fusiform face area (FFA) computes the static or invariant properties of faces. Here we directly tested this hypothesis by measuring the magnitude of response in each region to both dynamic and static stimuli. Consistent with the hypothesis, we found that the response to movies of faces was not significantly different from the response to static images of faces from these same movies in the right FFA and right occipital face area (OFA). By contrast the face-selective region in the right posterior STS (pSTS) responded nearly three times as strongly to dynamic faces as to static faces, and a face-selective region in the right anterior STS (aSTS) responded to dynamic faces only. Both of these regions also responded more strongly to moving faces than to moving bodies, indicating that they are preferentially engaged in processing dynamic information from faces, not in more general processing of any dynamic social stimuli. The response to dynamic and static faces was not significantly different in a third face-selective region in the posterior continuation of the STS (pcSTS). The strong selectivity of face-selective regions in the pSTS and aSTS, but not the FFA, OFA or pcSTS, for dynamic face information demonstrates a clear functional dissociation between different face-selective regions, and provides further clues into their function.     

Investigation of the human hippocampal formation using a randomized event-related paradigm and Z-shimmed functional MRI

Functional neuroimaging of the hippocampal formation has presented a challenge to neuroscientists because of the small size of the hippocampus proper and its location at the basal level of the brain. Choosing the appropriate control condition for subtraction-based studies has also proved difficult. Event-related experimental designs are a powerful tool in behavioral and electrophysiological studies. Recently, such experimental designs have been applied to functional MR imaging studies but these studies used large intertrial intervals in order to separate the slow blood flow response from temporally adjacent events, severely limiting the number of events that can be presented in a single run. This leads to poor statistical power and restrictions on the design of the experimental paradigm. We present data obtained using a rapidly presented, randomized event-related paradigm, combined with a novel fMRI imaging method designed to improve imaging in basal brain regions. The results demonstrate bilateral activation in the hippocampal formation in identification of novel complex scenes distinct from a learned basis set of complex scenes. Differential activation is obtained in the counter task of identifying a learned target complex scene against a background of novel scenes. The results are also compared with the more conventional block design complex scene paradigms previously reported by others. The block design provides strong posterior activation, likely related more to visual scene processing, whereas the event-related design provides more anterior hippocampal activation with the encoding of novel scenes.