Visual phonetic processing localized using speech and nonspeech face gestures in video and point‐light displays

The talking face affords multiple types of information. To isolate cortical sites with responsibility for integrating linguistically relevant visual speech cues, speech and nonspeech face gestures were presented in natural video and point‐light displays during fMRI scanning at 3.0T. Participants with normal hearing viewed the stimuli and also viewed localizers for the fusiform face area (FFA), the lateral occipital complex (LOC), and the visual motion (V5/MT) regions of interest (ROIs). The FFA, the LOC, and V5/MT were significantly less activated for speech relative to nonspeech and control stimuli. Distinct activation of the posterior superior temporal sulcus and the adjacent middle temporal gyrus to speech, independent of media, was obtained in group analyses. Individual analyses showed that speech and nonspeech stimuli were associated with adjacent but different activations, with the speech activations more anterior. We suggest that the speech activation area is the temporal visual speech area (TVSA), and that it can be localized with the combination of stimuli used in this study. Hum Brain Mapp, 2010. © 2010 Wiley‐Liss, Inc.     

Being BOLD: The neural dynamics of face perception

According to a non‐hierarchical view of human cortical face processing, selective responses to faces may emerge in a higher‐order area of the hierarchy, in the lateral part of the middle fusiform gyrus (fusiform face area [FFA]) independently from face‐selective responses in the lateral inferior occipital gyrus (occipital face area [OFA]), a lower order area. Here we provide a stringent test of this hypothesis by gradually revealing segmented face stimuli throughout strict linear descrambling of phase information [Ales et al., 2012]. Using a short sampling rate (500 ms) of fMRI acquisition and single subject statistical analysis, we show a face‐selective responses emerging earlier, that is, at a lower level of structural (i.e., phase) information, in the FFA compared with the OFA. In both regions, a face detection response emerging at a lower level of structural information for upright than inverted faces, both in the FFA and OFA, in line with behavioral responses and with previous findings of delayed responses to inverted faces with direct recordings of neural activity were also reported. Overall, these results support the non‐hierarchical view of human cortical face processing and open new perspectives for time‐resolved analysis at the single subject level of fMRI data obtained during continuously evolving visual stimulation. Hum Brain Mapp 38:120–139, 2017. © 2016 Wiley Periodicals, Inc.     

Facial color processing in the face‐selective regions: An fMRI study

Facial color is important information for social communication as it provides important clues to recognize a person's emotion and health condition. Our previous EEG study suggested that N170 at the left occipito‐temporal site is related to facial color processing (Nakajima et al., [2012]: Neuropsychologia 50:2499–2505). However, because of the low spatial resolution of EEG experiment, the brain region is involved in facial color processing remains controversial. In the present study, we examined the neural substrates of facial color processing using functional magnetic resonance imaging (fMRI). We measured brain activity from 25 subjects during the presentation of natural‐ and bluish‐colored face and their scrambled images. The bilateral fusiform face (FFA) area and occipital face area (OFA) were localized by the contrast of natural‐colored faces versus natural‐colored scrambled images. Moreover, region of interest (ROI) analysis showed that the left FFA was sensitive to facial color, whereas the right FFA and the right and left OFA were insensitive to facial color. In combination with our previous EEG results, these data suggest that the left FFA may play an important role in facial color processing. Hum Brain Mapp 35:4958–4964, 2014. © 2014 Wiley Periodicals, Inc.     

Maturational trajectories of local and long‐range functional connectivity in autism during face processing

Autism spectrum disorder (ASD) is characterized neurophysiologically by, among other things, functional connectivity abnormalities in the brain. Recent evidence suggests that the nature of these functional connectivity abnormalities might not be uniform throughout maturation. Comparing between adolescents and young adults (ages 14–21) with ASD and age‐ and IQ‐matched typically developing (TD) individuals, we previously documented, using magnetoencephalography (MEG) data, that local functional connectivity in the fusiform face areas (FFA) and long‐range functional connectivity between FFA and three higher order cortical areas were all reduced in ASD. Given the findings on abnormal maturation trajectories in ASD, we tested whether these results extend to preadolescent children (ages 7–13). We found that both local and long‐range functional connectivity were in fact normal in this younger age group in ASD. Combining the two age groups, we found that local and long‐range functional connectivity measures were positively correlated with age in TD, but negatively correlated with age in ASD. Last, we showed that local functional connectivity was the primary feature in predicting age in ASD group, but not in the TD group. Furthermore, local functional connectivity was only correlated with ASD severity in the older group. These results suggest that the direction of maturation of functional connectivity for processing of faces from childhood to young adulthood is itself abnormal in ASD, and that during the processing of faces, these trajectory abnormalities are more pronounced for local functional connectivity measures than they are for long‐range functional connectivity measures.     

Functional integration of the posterior superior temporal sulcus correlates with facial expression recognition

Face perception is essential for daily and social activities. Neuroimaging studies have revealed a distributed face network (FN) consisting of multiple regions that exhibit preferential responses to invariant or changeable facial information. However, our understanding about how these regions work collaboratively to facilitate facial information processing is limited. Here, we focused on changeable facial information processing, and investigated how the functional integration of the FN is related to the performance of facial expression recognition. To do so, we first defined the FN as voxels that responded more strongly to faces than objects, and then used a voxel‐based global brain connectivity method based on resting‐state fMRI to characterize the within‐network connectivity (WNC) of each voxel in the FN. By relating the WNC and performance in the “Reading the Mind in the Eyes” Test across participants, we found that individuals with stronger WNC in the right posterior superior temporal sulcus (rpSTS) were better at recognizing facial expressions. Further, the resting‐state functional connectivity (FC) between the rpSTS and right occipital face area (rOFA), early visual cortex (EVC), and bilateral STS were positively correlated with the ability of facial expression recognition, and the FCs of EVC‐pSTS and OFA‐pSTS contributed independently to facial expression recognition. In short, our study highlights the behavioral significance of intrinsic functional integration of the FN in facial expression processing, and provides evidence for the hub‐like role of the rpSTS for facial expression recognition. Hum Brain Mapp 37:1930–1940, 2016. © 2016 Wiley Periodicals, Inc.     

The relationship between repetition suppression and face perception

Repetition of identical face stimuli leads to fMRI response attenuation (fMRI adaptation, fMRIa) in the core face-selective occipito-temporal visual cortical network, involving the bilateral fusiform face area (FFA) and the occipital face area (OFA). However, the functional relevance of fMRIa observed in these regions is unclear as of today. Therefore, here we aimed at investigating the relationship between fMRIa and face perception ability by measuring in the same human participants both the repetition-induced reduction of fMRI responses and identity discrimination performance outside the scanner for upright and inverted face stimuli. In the correlation analysis, the behavioral and fMRI results for the inverted faces were used as covariates to control for the individual differences in overall object perception ability and basic visual feature adaptation processes, respectively. The results revealed a significant positive correlation between the participants’ identity discrimination performance and the strength of fMRIa in the core face processing network, but not in the extrastriate body area (EBA). Furthermore, we found a strong correlation of the fMRIa between OFA and FFA and also between OFA and EBA, but not between FFA and EBA. These findings suggest that there is a face-selective component of the repetition-induced reduction of fMRI responses within the core face processing network, which reflects functionally relevant adaptation processes involved in face identity perception.     

Coherent motion processing in autism spectrum disorder (ASD): An fMRI study

A deficit in global motion processing caused by a specific dysfunction of the visual dorsal pathway has been suggested to underlie perceptual abnormalities in subjects with autism spectrum disorders (ASD). However, the neural mechanisms associated with abnormal motion processing in ASD remain poorly understood. We investigated brain responses related to the detection of coherent and random motion in 15 male subjects with ASD and 15 age- and IQ-matched healthy controls (aged 13-19 years) using event-related functional magnetic resonance imaging (fMRI). Behaviorally, no significant group differences were observed between subjects with ASD and controls. Neurally, subjects with ASD showed increased brain activation in the left primary visual cortex across all conditions compared with controls. A significant interaction effect between group and condition was observed in the right superior parietal cortex resulting from increased neural activity in the coherent compared with the random motion conditions only in the control group. In addition, neural activity in area V5 was not differentially modulated by specific motion conditions in subjects with ASD. Functional connectivity analyses revealed positive correlations between the primary visual cortex and area V5 within both hemispheres, but no significant between-group differences in functional connectivity patterns along the dorsal stream. The data suggest that motion processing in ASD results in deviant activations in both the lower and higher processing stages of the dorsal pathway. This might reflect differences in the perception of visual stimuli in ASD, which possibly result in impaired integration of motion signals.     

The neural correlates of change detection in the face perception network

A common view is that visual processing within the ventral visual stream is modulated by attention and awareness. We used fMRI adaptation to investigate whether activation in a network of brain regions involved with face recognition--namely the fusiform face area (FFA), occipital face area (OFA) and right superior temporal sulcus (rSTS)--was modulated by physical changes to face stimuli or by observers' awareness of the changes. We sequentially presented two matrices of four faces. In two thirds of the trials one of the faces changed. We compared activations generated in three conditions (i) change detected trials, (ii) change blind trials, and (iii) no change trials. If face areas were sensitive to physical changes then we expected similar levels of activation for face changes regardless of change detection. If face areas were sensitive to levels of awareness of change then we expected greater levels of activation for detected changes compared to no change detection. We found that all three-face regions recovered from adaptation when subjects were aware of changes, but only OFA recovered from adaptation when subjects were not aware of the changes. These results suggest that within the face network OFA is involved in representing information that has not yet entered awareness and that consciousness is not an all-or-none phenomenon throughout the ventral stream.     

Amygdala–prefrontal cortex connectivity increased during face discrimination but not time perception

Time sensitivity is affected by emotional stimuli such as fearful faces. The effect of threatening stimuli on time perception depends on numerous factors, including task type and duration range. We applied a two‐interval forced‐choice task using face stimuli to healthy volunteers to evaluate time perception and emotion interaction using functional magnetic resonance imaging. We conducted finite impulse response analysis to examine time series for the significantly activated brain areas and psycho‐physical interaction to investigate the connectivity between selected regions. Time perception engaged a right‐lateralised frontoparietal network, while a face discrimination task activated the amygdala and fusiform face area (FFA). No voxels were active with regard to the effect of expression (fearful versus neutral). In parallel with this, our behavioural results showed that attending to the fearful faces did not cause duration overestimation. Finally, connectivity of the amygdala and FFA to the middle frontal gyrus increased during the face processing condition compared to the timing task. Overall, our results suggest that the prefrontal–amygdala connectivity might be required for the emotional processing of facial stimuli. On the other hand, attentional load, task type and task difficulty are discussed as possible factors that influence the effects of emotion on time perception.     

The fusiform face area is not sufficient for face recognition: evidence from a patient with dense prosopagnosia and no occipital face area

We tested functional activation for faces in patient D.F., who following acquired brain damage has a profound deficit in object recognition based on form (visual form agnosia) and also prosopagnosia that is undocumented to date. Functional imaging demonstrated that like our control observers, D.F. shows significantly more activation when passively viewing face compared to scene images in an area that is consistent with the fusiform face area (FFA) (p < 0.01). Control observers also show occipital face area (OFA) activation; however, whereas D.F.'s lesions appear to overlap the OFA bilaterally. We asked, given that D.F. shows FFA activation for faces, to what extent is she able to recognize faces? D.F. demonstrated a severe impairment in higher level face processing--she could not recognize face identity, gender or emotional expression. In contrast, she performed relatively normally on many face categorization tasks. D.F. can differentiate faces from non-faces given sufficient texture information and processing time, and she can do this is independent of color and illumination information. D.F. can use configural information for categorizing faces when they are presented in an upright but not a sideways orientation and given that she also cannot discriminate half-faces she may rely on a spatially symmetric feature arrangement. Faces appear to be a unique category, which she can classify even when she has no advance knowledge that she will be shown face images. Together, these imaging and behavioral data support the importance of the integrity of a complex network of regions for face identification, including more than just the FFA--in particular the OFA, a region believed to be associated with low-level processing.     

Location of human face‐selective cortex with respect to retinotopic areas

Functional Magnetic Resonance Imaging (fMRI) was used to identify a small area in the human posterior fusiform gyrus that responds selectively to faces (PF). In the same subjects, phase‐encoded rotating and expanding checkerboards were used with fMRI to identify the retinotopic visual areas V1, V2, V3, V3A, VP and V4v. PF was found to lie anterior to area V4v, with a small gap present between them. Further recordings in some of the same subjects used moving low‐contrast rings to identify the visual motion area MT. PF was found to lie ventral to MT. In addition, preliminary evidence was found using fMRI for a small area that responded to inanimate objects but not to faces in the collateral sulcus medial to PF. The retinotopic visual areas and MT responded equally to faces, control randomized stimuli, and objects. Weakly face‐selective responses were also found in ventrolateral occipitotemporal cortex anterior to V4v, as well as in the middle temporal gyrus anterior to MT. We conclude that the fusiform face area in humans lies in non‐retinotopic visual association cortex of the ventral form‐processing stream, in an area that may be roughly homologous in location to area TF or CITv in monkeys. Hum. Brain Mapping 7:29–37, 1999. © 1999 Wiley‐Liss, Inc.     

Functional plasticity in ventral temporal cortex following cognitive rehabilitation of a congenital prosopagnosic

We used functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) to measure neural changes associated with training configural processing in congenital prosopagnosia, a condition in which face identification abilities are not properly developed in the absence of brain injury or visual problems. We designed a task that required discriminating faces by their spatial configuration and, after extensive training, prosopagnosic MZ significantly improved at face identification. Event-related potential results revealed that although the N170 was not selective for faces before training, its selectivity after training was normal. fMRI demonstrated increased functional connectivity between ventral occipital temporal face-selective regions (right occipital face area and right fusiform face area) that accompanied improvement in face recognition. Several other regions showed fMRI activity changes with training; the majority of these regions increased connectivity with face-selective regions. Together, the neural mechanisms associated with face recognition improvements involved strengthening early face-selective mechanisms and increased coordination between face-selective and nonselective regions, particularly in the right hemisphere.     

Reading speech from still and moving faces: the neural substrates of visible speech

Speech is perceived both by ear and by eye. Unlike heard speech, some seen speech gestures can be captured in stilled image sequences. Previous studies have shown that in hearing people, natural time-varying silent seen speech can access the auditory cortex (left superior temporal regions). Using functional magnetic resonance imaging (fMRI), the present study explored the extent to which this circuitry was activated when seen speech was deprived of its time-varying characteristics. In the scanner, hearing participants were instructed to look for a prespecified visible speech target sequence ("voo" or "ahv") among other monosyllables. In one condition, the image sequence comprised a series of stilled key frames showing apical gestures (e.g., separate frames for "v" and "oo" [from the target] or "ee" and "m" [i.e., from nontarget syllables]). In the other condition, natural speech movement of the same overall segment duration was seen. In contrast to a baseline condition in which the letter "V" was superimposed on a resting face, stilled speech face images generated activation in posterior cortical regions associated with the perception of biological movement, despite the lack of apparent movement in the speech image sequence. Activation was also detected in traditional speech-processing regions including the left inferior frontal (Broca's) area, left superior temporal sulcus (STS), and left supramarginal gyrus (the dorsal aspect of Wernicke's area). Stilled speech sequences also generated activation in the ventral premotor cortex and anterior inferior parietal sulcus bilaterally. Moving faces generated significantly greater cortical activation than stilled face sequences, and in similar regions. However, a number of differences between stilled and moving speech were also observed. In the visual cortex, stilled faces generated relatively more activation in primary visual regions (V1/V2), while visual movement areas (V5/MT+) were activated to a greater extent by moving faces. Cortical regions activated more by naturally moving speaking faces included the auditory cortex (Brodmann's Areas 41/42; lateral parts of Heschl's gyrus) and the left STS and inferior frontal gyrus. Seen speech with normal time-varying characteristics appears to have preferential access to "purely" auditory processing regions specialized for language, possibly via acquired dynamic audiovisual integration mechanisms in STS. When seen speech lacks natural time-varying characteristics, access to speech-processing systems in the left temporal lobe may be achieved predominantly via action-based speech representations, realized in the ventral premotor cortex.     

Event‐related potential and functional MRI measures of face‐selectivity are highly correlated: A simultaneous ERP‐fMRI investigation

A face‐selective neural signal is reliably found in humans with functional MRI and event‐related potential (ERP) measures, which provide complementary information about the spatial and temporal properties of the neural response. However, because most neuroimaging studies so far have studied ERP and fMRI face‐selective markers separately, the relationship between them is still unknown. Here we simultaneously recorded fMRI and ERP responses to faces and chairs to examine the correlations across subjects between the magnitudes of fMRI and ERP face‐selectivity measures. Findings show that the face‐selective responses in the temporal lobe (i.e., fusiform gyrus—FFA) and superior temporal sulcus (fSTS), but not the face‐selective response in the occipital cortex (OFA), were highly correlated with the face‐selective N170 component. In contrast, the OFA was correlated with earlier ERPs at about 110 ms after stimulus‐onset. Importantly, these correlations reveal a temporal dissociation between the face‐selective area in the occipital lobe and face‐selective areas in the temporal lobe. Despite the very different time‐scale of the fMRI and EEG signals, our data show that a correlation analysis across subjects may be informative with respect to the latency in which different brain regions process information. Hum Brain Mapp, 2010. © 2010 Wiley‐Liss, Inc.     

Abnormal activation of the social brain during face perception in autism

ASD involves a fundamental impairment in processing social-communicative information from faces. Several recent studies have challenged earlier findings that individuals with autism spectrum disorder (ASD) have no activation of the fusiform gyrus (fusiform face area, FFA) when viewing faces. In this study, we examined activation to faces in the broader network of face-processing modules that comprise what is known as the social brain. Using 3T functional resonance imaging, we measured BOLD signal changes in 10 ASD subjects and 7 healthy controls passively viewing nonemotional faces. We replicated our original findings of significant activation of face identity-processing areas (FFA and inferior occipital gyrus, IOG) in ASD. However, in addition, we identified hypoactivation in a more widely distributed network of brain areas involved in face processing [including the right amygdala, inferior frontal cortex (IFC), superior temporal sulcus (STS), and face-related somatosensory and premotor cortex]. In ASD, we found functional correlations between a subgroup of areas in the social brain that belong to the mirror neuron system (IFC, STS) and other face-processing areas. The severity of the social symptoms measured by the Autism Diagnostic Observation Schedule was correlated with the right IFC cortical thickness and with functional activation in that area. When viewing faces, adults with ASD show atypical patterns of activation in regions forming the broader face-processing network and social brain, outside the core FFA and IOG regions. These patterns suggest that areas belonging to the mirror neuron system are involved in the face-processing disturbances in ASD.     

Gesturing tool use and tool transport actions modulates inferior parietal functional connectivity with the dorsal and ventral object processing pathways

Interacting with manipulable objects (tools) requires the integration of diverse computations supported by anatomically remote regions. Previous functional neuroimaging research has demonstrated the left supramarginal (SMG) exhibits functional connectivity to both ventral and dorsal pathways, supporting the integration of ventrally‐mediated tool properties and conceptual knowledge with dorsally‐computed volumetric and structural representations of tools. This architecture affords us the opportunity to test whether interactions between the left SMG, ventral visual pathway, and dorsal visual pathway are differentially modulated when participants plan and generate tool‐directed gestures emphasizing functional manipulation (tool use gesturing) or structure‐based grasping (tool transport gesturing). We found that functional connectivity between the left SMG, ventral temporal cortex (bilateral fusiform gyri), and dorsal visual pathway (left superior parietal lobule/posterior intraparietal sulcus) was maximal for tool transport planning and gesturing, whereas functional connectivity between the left SMG, left ventral anterior temporal lobe, and left frontal operculum was maximal for tool use planning and gesturing. These results demonstrate that functional connectivity to the left SMG is differentially modulated by tool use and tool transport gesturing, suggesting that distinct tool features computed by the two object processing pathways are integrated in the parietal lobe in the service of tool‐directed action.     

Aging and brain fitness (Commentary on Voelcker‐Rehage et al.)

The human extrastriate visual cortex contains functionally distinct regions where neuronal populations exhibit signals that are selective for objects. How such regions might play a causal role in underpinning our ability to recognize objects across different viewpoints remains uncertain. Here, we tested whether two extrastriate areas, the lateral occipital (LO) region and occipital face area (OFA), contained neuronal populations that play a causal role in recognizing two‐dimensional shapes across different rotations. We used visual priming to modulate the rotation‐sensitive activity of neuronal populations in these areas. State‐dependent transcranial magnetic stimulation (TMS) was applied after the presentation of a shape and immediately before a subsequent probe shape to which participants had to respond. We found that TMS applied to both the LO region and OFA modulated rotation‐invariant shape priming but, whereas the LO region was modulated by TMS for small rotations, the OFA was modulated for larger rotations. Importantly, our results demonstrate that a node in the face‐sensitive network, the OFA, participates in causally relevant encoding of non‐face stimuli.     

Evaluating the organizational structure and specificity of network topology within the face processing system

There is increasing appreciation that network‐level interactions among regions produce components of face processing previously ascribed to individual regions. Our goals were to use an exhaustive data‐driven approach to derive and quantify the topology of directed functional connections within a priori defined nodes of the face processing network and evaluate whether the topology is category‐specific. Young adults were scanned with fMRI as they viewed movies of faces, objects, and scenes. We employed GIMME to model effective connectivity among core and extended face processing regions, which allowed us to evaluate all possible directional connections, under each viewing condition (face, object, place). During face processing, we observed directional connections from the right posterior superior temporal sulcus to both the right occipital face area and right fusiform face area (FFA), which does not reflect the topology reported in prior studies. We observed connectivity between core and extended regions during face processing, but this limited to a feed‐forward connection from the FFA to the amygdala. Finally, the topology of connections was unique to face processing. These findings suggest that the pattern of directed functional connections within the face processing network, particularly in the right core regions, may not be as hierarchical and feed‐forward as described previously. Our findings support the notion that topologies of network connections are specialized, emergent, and dynamically responsive to task demands.     

Linearity of the fMRI response in category‐selective regions of human visual cortex

The goal of this study was to determine the linearity of the blood oxygen level‐dependent (BOLD) response, as measured by functional magnetic resonance imaging (fMRI), in category‐selective regions of human visual cortex. We defined regions of the temporal lobe that were selective to faces (fusiform face area, FFA) and places (parahippocampal place area, PPA). We then determined the linearity of the BOLD response in these regions to their preferred and nonpreferred stimuli. First, we tested the principle of scaling. As we increased the visibility of the stimulus, there was a corresponding linear increase in the fMRI signal in the FFA and PPA to their preferred stimulus (face and place, respectively). In contrast, responses in the FFA and PPA to the nonpreferred stimulus did not conform to the principle of scaling. Next, we asked whether the fMRI response in these regions of visual cortex conformed to the principle of additivity. To assess this, we determined whether the response to a long stimulus block could be predicted by adding the response to multiple shorter duration blocks. Although the fMRI response in the FFA and PPA was generally linear to the preferred stimulus, a more nonlinear response was apparent to the nonpreferred stimulus. In conclusion, the linearity of the BOLD response in the human ventral visual pathway varied across cortical region and stimulus category. This suggests that measures of linearity may provide a useful indication of neural selectivity in the brain. Hum Brain Mapp, 2009. © 2008 Wiley‐Liss, Inc.     

Stria terminalis,amygdala, and temporoparietal junction networks facilitate efficient emotion processing under expectations

Rapid emotion processing is an ecologically essential ability for survival in social environments in which threatening or advantageous encounters dynamically and rapidly occur. Efficient emotion recognition is subserved by different processes, depending on one's expectations; however, the underlying functional and structural circuitry is still poorly understood. In this study, we delineate brain networks that subserve fast recognition of emotion in situations either congruent or incongruent with prior expectations. For this purpose, we used multimodal neuroimaging and investigated performance on a dynamic emotion perception task. We show that the extended amygdala structural and functional networks relate to speed of emotion processing under threatening conditions. Specifically, increased microstructure of the right stria terminalis, an amygdala white‐matter pathway, was related to faster detection of emotion during actual presentation of anger or after cueing anger. Moreover, functional connectivity of right amygdala with limbic regions was related to faster detection of anger congruent with cue, suggesting selective attention to threat. On the contrary, we found that faster detection of anger incongruent with cue engaged the ventral attention “reorienting” network. Faster detection of happiness, in either expectancy context, engaged a widespread frontotemporal‐subcortical functional network. These findings shed light on the functional and structural circuitries that facilitate speed of emotion recognition and, for the first time, elucidate a role for the stria terminalis in human emotion processing.