Shared neural codes for visual and semantic information about familiar faces in a common representational space

Processes evoked by seeing a personally familiar face encompass recognition of visual appearance and activation of social and person knowledge. Whereas visual appearance is the same for all viewers, social and person knowledge may be more idiosyncratic. Using between-subject multivariate decoding of hyperaligned functional magnetic resonance imaging data, we investigated whether representations of personally familiar faces in different parts of the distributed neural system for face perception are shared across individuals who know the same people. We found that the identities of both personally familiar and merely visually familiar faces were decoded accurately across brains in the core system for visual processing, but only the identities of personally familiar faces could be decoded across brains in the extended system for processing nonvisual information associated with faces. Our results show that personal interactions with the same individuals lead to shared neural representations of both the seen and unseen features that distinguish their identities.

Face recognition is essential for effective social interactions. When we see a familiar face, we spontaneously retrieve person knowledge and the position occupied by that familiar individual in our social network. This information sets us up for the most appropriate behavior with that specific individual. The importance of familiar faces for social interactions is reflected in the way the human brain processes these stimuli. Familiar faces are processed in a prioritized way with faster detection even in suboptimal conditions (1, 2). Familiarity associated with faces warps their visual representation (3) and can result in a more homogenous representation across the visual field in areas with retinotopic organization (4). Recognition of familiar faces entails processing not only their visual appearance but also retrieval of person knowledge and an emotional response (5–10). Different parts of the distributed neural system for face perception contribute to these processes (5, 10, 11). The core system for face perception processes visual appearance, resulting in view-invariant representations of identity in anterior temporal and inferior frontal face areas (12–14). The extended system for face perception plays a role in extracting semantic information from faces as well as emotional responses (5, 11, 15, 16).Here, we investigated the neural codes for high-level visual and semantic information about personally familiar faces. Specifically, we asked whether these codes are supported by a common set of basis functions that are shared across people who are personally familiar with the same individuals. We measured patterns of brain activity with functional magnetic resonance imaging (fMRI) while participants viewed images of personally familiar faces and faces of strangers who were only visually familiar. We used hyperalignment to derive a set of basis functions that align brain response patterns in a common, high-dimensional information space (17–19). Hyperalignment transformation parameters were based on participants’ brain activity measured while watching The Grand Budapest Hotel (20), an engaging comedy-drama with rich characterizations of several individuals. We found that these basis functions capture shared representations of visual appearance in the core system for both personally familiar faces and visually familiar faces of strangers. Surprisingly, we also found basis functions that capture shared representations of personally familiar others, but not visually familiar strangers, in extended system areas that are associated with representation of person knowledge, theory of mind, and emotion. Importantly, these basis functions are derived from brain responses to the movie and are, thus, not specific to the familiar individuals whose faces were the experimental stimuli. These results show that the face processing system encodes both visual and nonvisual high-level semantic information about personally familiar others in a neural information space that is not specific to a given set of faces and that is shared across brains.