Decoding the representation of numerical values from brain activation patterns

Human neuroimaging studies have increasingly converged on the possibility that the neural representation of specific numbers may be decodable from brain activity, particularly in parietal cortex. Multivariate machine learning techniques have recently demonstrated that the neural representation of individual concrete nouns can be decoded from fMRI patterns, and that some patterns are general over people. Here we use these techniques to investigate whether the neural codes for quantities of objects can be accurately decoded. The pictorial mode (nonsymbolic) depicted a set of objects pictorially (e.g., a picture of three tomatoes), whereas the digit‐object mode depicted quantities as combination of a digit (e.g., 3) with a picture of a single object. The study demonstrated that quantities of objects were decodable from neural activation patterns, in parietal regions. These brain activation patterns corresponding to a given quantity were common across objects and across participants in the pictorial mode. Other important findings included better identification of individual numbers in the pictorial mode, partial commonality of neural patterns across the two modes, and hemispheric asymmetry with pictorially‐depicted numbers represented bilaterally and numbers in the digit‐object mode represented primarily in the left parietal regions. The findings demonstrate the ability to identify individual quantities of objects based on neural patterns, indicating the presence of stable neural representations of numbers. Additionally, they indicate a predominance of neural representation of pictorially depicted numbers over the digit‐object mode. Hum Brain Mapp 34:2624–2634, 2013. © 2012 Wiley Periodicals, Inc.     

Predicting vocal emotion expressions from the human brain

Speech is an important carrier of emotional information. However, little is known about how different vocal emotion expressions are recognized in a receiver's brain. We used multivariate pattern analysis of functional magnetic resonance imaging data to investigate to which degree distinct vocal emotion expressions are represented in the receiver's local brain activity patterns. Specific vocal emotion expressions are encoded in a right fronto‐operculo‐temporal network involving temporal regions known to subserve suprasegmental acoustic processes and a fronto‐opercular region known to support emotional evaluation, and, moreover, in left temporo‐cerebellar regions covering sequential processes. The right inferior frontal region, in particular, was found to differentiate distinct emotional expressions. The present analysis reveals vocal emotion to be encoded in a shared cortical network reflected by distinct brain activity patterns. These results shed new light on theoretical and empirical controversies about the perception of distinct vocal emotion expressions at the level of large‐scale human brain signals. Hum Brain Mapp, 2013. © 2012 Wiley Periodicals, Inc.     

Neuronal representation of occluded objects in the human brain

Occluding surfaces frequently obstruct the object of interest yet are easily dealt with by the visual system. Here, we test whether neural areas known to participate in motion perception and eye movements are regions that also process occluded motion. Functional magnetic resonance imaging (fMRI) was used to assess brain activation while subjects watched a moving ball become occluded. Areas activated during occluded motion included the intraparietal sulcus (IPS) as well as middle temporal (MT) regions analogous to monkey MT/MST. A second experiment showed that these results were not due to motor activity. These findings suggest that human cortical regions involved in perceiving occluded motion are similar to regions that process real motion and regions responsible for eye movements. The intraparietal sulcus may be involved in predicting the location of an unseen target for future hand or eye movements.     

The representation of mental state information in schizophrenia and first-degree relatives: a multivariate pattern analysis of fMRI data

Individuals with a schizophrenia-spectrum disorder (SSD) and those at familial high risk (FHR) for SSDs experience social difficulties that are related to neural abnormalities in the network of brain regions recruited during theory of mind (ToM). Prior work with these groups has focused almost exclusively on characterizing the involvement of these regions in ToM. Here, we examine the representational content of these regions using multivariate pattern analysis. We analyzed two previously collected datasets of SSD, FHR and control participants who, while undergoing functional magnetic resonance imaging, completed the false-belief task in which they read stories describing beliefs or physical representations (e.g. photographs). Univariate and multivariate analyses were performed in regions of interest to evaluate group differences in task-based activation and representational content, respectively. Compared to non-SSDs, SSDs showed reduced decoding accuracy for the category of mental states in the right temporo-parietal junction—which was related to false-belief accuracy—and the dorsal medial prefrontal cortex (DMPFC) and reduced involvement of DMPFC for mental state understanding. FHR showed no differences in decoding accuracy or involvement compared to non-FHR. Given prior studies of disrupted neural involvement in FHR and the lack of decoding differences observed here, the onset of illness may involve processes that corrupt how mental state information is represented.     

The timing and precision of action prediction in the aging brain

Successful social interactions depend on the ability to anticipate other people's actions. Current conceptualizations of brain function propose that causes of sensory input are inferred through their integration with internal predictions generated in the observer's motor system during action observation. Less is known concerning how action prediction changes with age. Previously we showed that internal action representations are less specific in older compared with younger adults at behavioral and neural levels. Here, we characterize how neural activity varies while healthy older adults aged 56–71 years predict the time‐course of an unfolding action as well as the relation to task performance. By using fMRI, brain activity was measured while participants observed partly occluded actions and judged the temporal coherence of the action continuation that was manipulated. We found that neural activity in frontoparietal and occipitotemporal regions increased the more an action continuation was shifted backwards in time. Action continuations that were shifted towards the future preferentially engaged early visual cortices. Increasing age was associated with neural activity that extended from posterior to anterior regions in frontal and superior temporal cortices. Lower sensitivity in action prediction resulted in activity increases in the caudate. These results imply that the neural implementation of predicting actions undergoes similar changes as the neural process of executing actions in older adults. The comparison between internal predictions and sensory input seems to become less precise with age leading to difficulties in anticipating observed actions accurately, possibly due to less specific internal action models. Hum Brain Mapp 37:54–66, 2016. © 2015 The Authors Human Brain Mapping Published by Wiley Periodicals, Inc.     

Recent developments in multivariate pattern analysis for functional MRI

Multivariate pattern analysis(MVPA) is a recently-developed approach for functional magnetic resonance imaging(fMRI) data analyses.Compared with the traditional univariate methods,MVPA is more sensitive to subtle changes in multivariate patterns in fMRI data.In this review,we introduce several significant advances in MVPA applications and summarize various combinations of algorithms and parameters in different problem settings.The limitations of MVPA and some critical questions that need to be addressed in future research are also discussed.     

Modality‐independent representations of small quantities based on brain activation patterns

Machine learning or MVPA (Multi Voxel Pattern Analysis) studies have shown that the neural representation of quantities of objects can be decoded from fMRI patterns, in cases where the quantities were visually displayed. Here we apply these techniques to investigate whether neural representations of quantities depicted in one modality (say, visual) can be decoded from brain activation patterns evoked by quantities depicted in the other modality (say, auditory). The main finding demonstrated, for the first time, that quantities of dots were decodable by a classifier that was trained on the neural patterns evoked by quantities of auditory tones, and vice‐versa. The representations that were common across modalities were mainly right‐lateralized in frontal and parietal regions. A second finding was that the neural patterns in parietal cortex that represent quantities were common across participants. These findings demonstrate a common neuronal foundation for the representation of quantities across sensory modalities and participants and provide insight into the role of parietal cortex in the representation of quantity information. Hum Brain Mapp 37:1296‐1307, 2016. © 2016 Wiley Periodicals, Inc.     

Bilateral cortical representation of the trunk midline in human first somatic sensory area

The cortical representation of the trunk zone in the human first somatosensory area was studied with functional magnetic resonance imaging (fMRI) to establish whether the cutaneous regions close to the midline are represented in this area of both hemispheres. Cortical activation foci evoked by unilateral tactile stimulation of ventral trunk regions were detected in the postcentral gyrus of the contralateral hemisphere slightly medial to or just behind the omega-shaped region of the central sulcus and in the anterior bank of the postcentral sulcus. These regions probably correspond to the trunk ventral midline representation zones of areas 3a-3b and 1-2, respectively. Stimulation of cutaneous regions adjacent to the midline evoked activation foci also in the ipsilateral postcentral gyrus in regions symmetrical to those activated in the contralateral hemisphere. These data demonstrate that in humans, as in nonhuman primates, the cutaneous regions adjacent to the trunk midline are represented bilaterally in the first somatic sensory cortex. Whether the ipsilateral activation depends on callosal or extracallosal inputs remains to be elucidated.     

Effective functional mapping of fMRI data with support‐vector machines

There is a growing interest in using support vector machines (SVMs) to classify and analyze fMRI signals, leading to a wide variety of applications ranging from brain state decoding to functional mapping of spatially and temporally distributed brain activations. Studies so far have generated functional maps using the vector of weight values generated by the SVM classification process, or alternatively by mapping the correlation coefficient between the fMRI signal at each voxel and the brain state determined by the SVM. However, these approaches are limited as they do not incorporate both the information involved in the SVM prediction of a brain state, namely, the BOLD activation at voxels and the degree of involvement of different voxels as indicated by their weight values. An important implication of the above point is that two different datasets of BOLD signals, presumably obtained from two different experiments, can potentially produce two identical hyperplanes irrespective of their differences in data distribution. Yet, the two sets of signal inputs could correspond to different functional maps. With this consideration, we propose a new method called Effect Mapping that is generated as a product of the weight vector and a newly computed vector of mutual information between BOLD activations at each voxel and the SVM output. By applying this method on neuroimaging data of overt motor execution in nine healthy volunteers, we demonstrate higher decoding accuracy indicating the greater efficacy of this method. Hum Brain Mapp, 2010. © 2010 Wiley‐Liss, Inc.     

Representations of modality‐specific affective processing for visual and auditory stimuli derived from functional magnetic resonance imaging data

There is converging evidence that people rapidly and automatically encode affective dimensions of objects, events, and environments that they encounter in the normal course of their daily routines. An important research question is whether affective representations differ with sensory modality. This research examined the nature of the dependency of affect and sensory modality at a whole‐brain level of analysis in an incidental affective processing paradigm. Participants were presented with picture and sound stimuli that differed in positive or negative valence in an event‐related functional magnetic resonance imaging experiment. Global statistical tests, applied at a level of the individual, demonstrated significant sensitivity to valence within modality, but not valence across modalities. Modality‐general and modality‐specific valence hypotheses predict distinctly different multidimensional patterns of the stimulus conditions. Examination of lower dimensional representation of the data demonstrated separable dimensions for valence processing within each modality. These results provide support for modality‐specific valence processing in an incidental affective processing paradigm at a whole‐brain level of analysis. Future research should further investigate how stimulus‐specific emotional decoding may be mediated by the physical properties of the stimuli. Hum Brain Mapp 35:3558–3568, 2014. © 2013 Wiley Periodicals, Inc .     

Assessing paedophilia based on the haemodynamic brain response to face images

Objectives: Objective assessment of sexual preferences may be of relevance in the treatment and prognosis of child sexual offenders. Previous research has indicated that this can be achieved by pattern classification of brain responses to sexual child and adult images. Our recent research showed that human face processing is tuned to sexual age preferences. This observation prompted us to test whether paedophilia can be inferred based on the haemodynamic brain responses to adult and child faces. Methods: Twenty-four men sexually attracted to prepubescent boys or girls (paedophiles) and 32 men sexually attracted to men or women (teleiophiles) were exposed to images of child and adult, male and female faces during a functional magnetic resonance imaging (fMRI) session. Results: A cross-validated, automatic pattern classification algorithm of brain responses to facial stimuli yielded four misclassified participants (three false positives), corresponding to a specificity of 91% and a sensitivity of 95%. Conclusions: These results indicate that the functional response to facial stimuli can be reliably used for fMRI-based classification of paedophilia, bypassing the problem of showing child sexual stimuli to paedophiles.     

Primary auditory cortex representation of fear‐conditioned musical sounds

Auditory cortex is required for discriminative fear conditioning beyond the classical amygdala microcircuit, but its precise role is unknown. It has previously been suggested that Heschl's gyrus, which includes primary auditory cortex (A1), but also other auditory areas, encodes threat predictions during presentation of conditioned stimuli (CS) consisting of monophones, or frequency sweeps. The latter resemble natural prosody and contain discriminative spectro‐temporal information. Here, we use functional magnetic resonance imaging (fMRI) in humans to address CS encoding in A1 for stimuli that contain only spectral but no temporal discriminative information. Two musical chords (complex) or two monophone tones (simple) were presented in a signaled reinforcement context (reinforced CS+ and nonreinforced CS?), or in a different context without reinforcement (neutral sounds, NS1 and NS2), with an incidental sound detection task. CS/US association encoding was quantified by the increased discriminability of BOLD patterns evoked by CS+/CS?, compared to NS pairs with similar physical stimulus differences and task demands. A1 was defined on a single‐participant level and based on individual anatomy. We find that in A1, discriminability of CS+/CS? was higher than for NS1/NS2. This representation of unconditioned stimulus (US) prediction was of comparable magnitude for both types of sounds. We did not observe such encoding outside A1. Different from frequency sweeps investigated previously, musical chords did not share representations of US prediction with monophone sounds. To summarize, our findings suggest decodable representation of US predictions in A1, for various types of CS, including musical chords that contain no temporal discriminative information.     

Exploring commonalities across participants in the neural representation of objects

The question of whether the neural encodings of objects are similar across different people is one of the key questions in cognitive neuroscience. This article examines the commonalities in the internal representation of objects, as measured with fMRI, across individuals in two complementary ways. First, we examine the commonalities in the internal representation of objects across people at the level of interobject distances, derived from whole brain fMRI data, and second, at the level of spatially localized anatomical brain regions that contain sufficient information for identification of object categories, without making the assumption that their voxel patterns are spatially matched in a common space. We examine the commonalities in internal representation of objects on 3T fMRI data collected while participants viewed line drawings depicting various tools and dwellings. This exploratory study revealed the extent to which the representation of individual concepts, and their mutual similarity, is shared across participants.     

Statistical learning of temporal community structure in the hippocampus

The hippocampus is involved in the learning and representation of temporal statistics, but little is understood about the kinds of statistics it can uncover. Prior studies have tested various forms of structure that can be learned by tracking the strength of transition probabilities between adjacent items in a sequence. We test whether the hippocampus can learn higher‐order structure using sequences that have no variance in transition probability and instead exhibit temporal community structure. We find that the hippocampus is indeed sensitive to this form of structure, as revealed by its representations, activity dynamics, and connectivity with other regions. These findings suggest that the hippocampus is a sophisticated learner of environmental regularities, able to uncover higher‐order structure that requires sensitivity to overlapping associations. © 2015 Wiley Periodicals, Inc.     

Neural representation of emotion regulation goals

The use of top–down cognitive control mechanisms to regulate emotional responses as circumstances change is critical for mental and physical health. Several theoretical models of emotion regulation have been postulated; it remains unclear, however, in which brain regions emotion regulation goals (e.g., the downregulation of fear) are represented. Here, we examined the neural mechanisms of regulating emotion using fMRI and identified brain regions representing reappraisal goals. Using a multimethodological analysis approach, combining standard activation‐based and pattern‐information analyses, we identified a distributed network of lateral frontal, temporal, and parietal regions implicated in reappraisal and within it, a core system that represents reappraisal goals in an abstract, stimulus‐independent fashion. Within this core system, the neural pattern‐separability in a subset of regions including the left inferior frontal gyrus, middle temporal gyrus, and inferior parietal lobe was related to the success in emotion regulation. Those brain regions might link the prefrontal control regions with the subcortical affective regions. Given the strong association of this subsystem with inner speech functions and semantic memory, we conclude that those cognitive mechanisms may be used for orchestrating emotion regulation. Hum Brain Mapp 37:600–620, 2016. © 2015 Wiley Periodicals, Inc.     

Decoding the individual finger movements from single‐trial functional magnetic resonance imaging recordings of human brain activity

Multivariate pattern classification analysis (MVPA) has been applied to functional magnetic resonance imaging (fMRI) data to decode brain states from spatially distributed activation patterns. Decoding upper limb movements from non‐invasively recorded human brain activation is crucial for implementing a brain–machine interface that directly harnesses an individual's thoughts to control external devices or computers. The aim of this study was to decode the individual finger movements from fMRI single‐trial data. Thirteen healthy human subjects participated in a visually cued delayed finger movement task, and only one slight button press was performed in each trial. Using MVPA, the decoding accuracy (DA) was computed separately for the different motor‐related regions of interest. For the construction of feature vectors, the feature vectors from two successive volumes in the image series for a trial were concatenated. With these spatial–temporal feature vectors, we obtained a 63.1% average DA (84.7% for the best subject) for the contralateral primary somatosensory cortex and a 46.0% average DA (71.0% for the best subject) for the contralateral primary motor cortex; both of these values were significantly above the chance level (20%). In addition, we implemented searchlight MVPA to search for informative regions in an unbiased manner across the whole brain. Furthermore, by applying searchlight MVPA to each volume of a trial, we visually demonstrated the information for decoding, both spatially and temporally. The results suggest that the non‐invasive fMRI technique may provide informative features for decoding individual finger movements and the potential of developing an fMRI‐based brain–machine interface for finger movement.     

Intrinsic brain activity as a diagnostic biomarker in children with benign epilepsy with centrotemporal spikes

Benign epilepsy with centrotemporal spikes (BECTS) is often associated with neural circuit dysfunction, particularly during the transient active state characterized by interictal epileptiform discharges (IEDs). Little is known, however, about the functional neural circuit abnormalities in BECTS without IEDs, or if such abnormalities could be used to differentiate BECTS patients without IEDs from healthy controls (HCs) for early diagnosis. To this end, we conducted resting‐state functional magnetic resonance imaging (RS‐fMRI) and simultaneous Electroencephalogram (EEG) in children with BECTS (n = 43) and age‐matched HC (n = 28). The simultaneous EEG recordings distinguished BECTS with IEDs (n = 20) from without IEDs (n = 23). Intrinsic brain activity was measured in all three groups using the amplitude of low frequency fluctuation at rest. Compared to HC, BECTS patients with IEDs exhibited an intrinsic activity abnormality in the thalamus, suggesting that thalamic dysfunction could contribute to IED emergence while patients without IEDs exhibited intrinsic activity abnormalities in middle frontal gyrus and superior parietal gyrus. Using multivariate pattern classification analysis, we were able to differentiate BECTS without IEDs from HCs with 88.23% accuracy. BECTS without epileptic transients can be distinguished from HC and BECTS with IEDs by unique regional abnormalities in resting brain activity. Both transient abnormalities as reflected by IEDs and chronic abnormalities as reflected by RS‐fMRI may contribute to BECTS development and expression. Intrinsic brain activity and multivariate pattern classification techniques are promising tools to diagnose and differentiate BECTS syndromes. Hum Brain Mapp 36:3878–3889, 2015. © 2015 Wiley Periodicals, Inc.     

Does fMRI repetition suppression reveal mirror neuron activity in the human brain? Insights from univariate and multivariate analysis

Mirror neurons (MN) have been proposed as the neural substrate for a wide range of clinical, social and cognitive phenomena. Over the last decade, a commonly used tool for investigating MN activity in the human brain has been functional magnetic resonance (fMRI) repetition suppression (RS) paradigms. However, the available evidence is mixed, largely owing to inconsistent application of the methodological criteria necessary to infer MN properties. This raises concerns about the degree to which one can infer the presence (or absence) of MN activity from earlier accounts that adopted RS paradigms. We aimed to clarify this issue using a well‐validated fMRI RS paradigm and tested for mirror properties by rigorously applying the widely accepted criteria necessary to demonstrate MN activity using traditional univariate techniques and Multivariate Pattern Analysis (MVPA). While univariate whole brain analysis in healthy adults showed uni‐modal RS effects within the supplementary motor area, no evidence for cross‐modal RS effects consistent with mirror neuron activity was found. MVPA on the other hand revealed a region along the anterior intraparietal sulcus that met the criteria for MN activity. Taken together, these results clarify disparate evidence from earlier RS studies, highlighting that traditional univariate analysis of RS data may not be sensitive for detecting MN activity when rigorously applying the requisite criteria. In light of these findings, we recommend that short of increasing sample sizes substantially, future studies using RS paradigms to investigate MNs across the human brain consider the use of MVPA.