Mechanisms of hierarchical reinforcement learning in cortico-striatal circuits 2: evidence from fMRI

The frontal lobes may be organized hierarchically such that more rostral frontal regions modulate cognitive control operations in caudal regions. In our companion paper (Frank MJ, Badre D. 2011. Mechanisms of hierarchical reinforcement learning in corticostriatal circuits I: computational analysis. 22:509-526), we provide novel neural circuit and algorithmic models of hierarchical cognitive control in cortico-striatal circuits. Here, we test key model predictions using functional magnetic resonance imaging (fMRI). Our neural circuit model proposes that contextual representations in rostral frontal cortex influence the striatal gating of contextual representations in caudal frontal cortex. Reinforcement learning operates at each level, such that the system adaptively learns to gate higher order contextual information into rostral regions. Our algorithmic Bayesian "mixture of experts" model captures the key computations of this neural model and provides trial-by-trial estimates of the learner's latent hypothesis states. In the present paper, we used these quantitative estimates to reanalyze fMRI data from a hierarchical reinforcement learning task reported in Badre D, Kayser AS, D'Esposito M. 2010. Frontal cortex and the discovery of abstract action rules. Neuron. 66:315--326. Results validate key predictions of the models and provide evidence for an individual cortico-striatal circuit for reinforcement learning of hierarchical structure at a specific level of policy abstraction. These findings are initially consistent with the proposal that hierarchical control in frontal cortex may emerge from interactions among nested cortico-striatal circuits at different levels of abstraction.     

On the benefits of not trying: brain activity and connectivity reflecting the interactions of explicit and implicit sequence learning

Under certain circumstances, implicit, automatic learning may be attenuated by explicit memory processes. We explored the brain basis of this phenomenon in a functional magnetic resonance imaging (fMRI) study of motor sequence learning. Using a factorial design that crossed subjective intention to learn (explicit versus implicit) with sequence difficulty (a standard versus a more complex alternating sequence), we show that explicit attempts to learn the difficult sequence produce a failure of implicit learning and, in a follow-up behavioural experiment, that this failure represents a suppression of learning itself rather than of the expression of learning. This suppression is associated with sustained right frontal activation and attenuation of learning-related changes in the medial temporal lobe and the thalamus. Furthermore, this condition is characterized by a reversal of the fronto-thalamic connectivity observed with unimpaired implicit learning. The findings demonstrate a neural basis for a well-known behavioural effect: the deleterious impact of an explicit search upon implicit learning.     

Learning-related neuronal responses in prefrontal cortex studied with functional neuroimaging

We assessed time-dependent neuronal activity accompanying learning using functional magnetic resonance imaging (fMRI). An artificial grammar learning paradigm enabled us to dissociate activations associated with individual item learning from those involved in learning the underlying grammar system. We show that a localized region of right prefrontal cortex (PFC) is preferentially sensitive to individual item learning during the early stages of the experiment, while the left PFC region is sensitive to grammar learning which occurred across the entire course of the experiment. In addition to dissociating these two types of learning, we were able to characterize the effect of rule acquisition on neuronal responses associated with explicit learning of individual items. This effect was expressed as modulation of the time-dependent right PFC activations such that the early increase in activation associated with item learning was attenuated as the experiment progressed. In a further analysis we used structural equation modelling to explore time-dependent changes in inter-regional connectivity as a function of both item and grammar rule learning. Although there were no significant effects of item learning on the measured path strengths, rule learning was associated with a decrease in right fronto-parietal connectivity and an increase in connectivity between left and right PFC. Further fronto-parietal path strengths were observed to change, with an increase in left fronto-parietal and a decrease in right fronto-parietal connectivity path strength from right PFC to left parietal cortex. We interpret our findings in terms of a left frontal system mediating the semantic analysis of study items and directly influencing a right fronto-parietal system associated with episodic memory retrieval.     

Prefrontal-hippocampal dynamics involved in learning regularities across episodes

Using functional magnetic resonance imaging, the neural correlates of context-specific memories and invariant memories about regularities across episodes were investigated. Volunteers had to learn conjunctions between objects and positions. In an invariant learning condition, positions were held constant, enabling subjects to learn regularities across trials. By contrast, in a context-specific condition object-position conjunctions were trial unique. Performance increase in the invariant learning condition was paralleled by a learning-related increase of inferior frontal gyrus activation and ventral striatal activation and a decrease of hippocampus activation. Conversely, in the context-specific condition hippocampal activation was constant across trials. We argue that the learning-related hippocampal activation pattern might be due to reduced relational binding requirements once regularities are extracted. Furthermore, we propose that the learning-related prefrontal modulation reflects the requirement to extract and maintain regularities across trials and the adjustment of object-position conjunctions on the basis of the extracted knowledge. Finally, our data suggest that the ventral striatum encodes the increased predictability of spatial features as a function of learning. Taken together, these results indicate a transition of the relative roles of distinct brain regions during learning regularities across multiple episodes: regularity learning is characterized by a shift from a hippocampal to a prefrontal-striatal brain system.     

Cerebral bases of subliminal and supraliminal priming during reading

Several studies have investigated the neural correlates of conscious perception by contrasting functional magnetic resonance imaging (fMRI) activation to conscious and nonconscious visual stimuli. The results often reveal an amplification of posterior occipito-temporal activation and its extension into a parieto-frontal network. However, some of these effects might be due to a greater deployment of attentional or strategical processes in the conscious condition. Here, we examined the brain activity evoked by visible and invisible stimuli, both of which were irrelevant to the task. We collected fMRI data in a masking paradigm in which subliminal versus supraliminal letter strings were presented as primes while subjects focused attention on another subsequent, highly visible target word. Under those conditions, prime visibility was associated with greater activity confined to bilateral posterior occipito-temporal cortices, without extension into frontal and parietal cortices. However, supraliminal primes, compared with subliminal primes, evoked more extensive repetition suppression in a widely distributed set of parieto-frontal areas. Furthermore, only supraliminal primes caused phonological repetition enhancement in left inferior frontal and anterior insular cortex. Those results suggest a 2-stage view of conscious access: Relative to masked stimuli, unmasked stimuli elicit increased occipito-temporal activity, thus allowing them to compete for global conscious access and to induce priming in multiple distant areas. In the absence of attention, however, their access to a second stage of distributed parieto-frontal processing may remain blocked.     

Disruption of functional connectivity of the default-mode network in alcoholism

The default mode network (DMN) comprises brain structures maximally active at rest. Disturbance of network nodes or their connections occurs with some neuropsychiatric conditions and may underlie associated dysfunction. DMN connectivity has not been examined in alcoholism, which is marked by compromised DMN nodes and impaired spatial working memory. To test whether performance would be related to DMN integrity, we examined DMN functional connectivity using functional magnetic resonance imaging (fMRI) data and graph theory analysis. We assumed that disruption of short paths between network nodes would attenuate processing efficiency. Alcoholics and controls were scanned at rest and during a spatial working memory task. At rest, the spontaneous slow fluctuations of fMRI signals in the posterior cingulate and cerebellar regions in alcoholics were less synchronized than in controls, indicative of compromised functional connectivity. Graph theory analysis indicated that during rest, alcoholics had significantly lower efficiency indices than controls between the posterior cingulate seed and multiple cerebellar sites. Greater efficiency in several connections correlated with longer sobriety in alcoholics. During the task, on which alcoholics performed on par with controls, connectivity between the left posterior cingulate seed and left cerebellar regions was more robust in alcoholics than controls and suggests compensatory networking to achieve normal performance.     

Dissociating the roles of the cerebellum and motor cortex during adaptive learning: the motor cortex retains what the cerebellum learns

Adaptation to a novel visuomotor transformation has revealed important principles regarding learning and memory. Computational and behavioral studies have suggested that acquisition and retention of a new visuomotor transformation are distinct processes. However, this dissociation has never been clearly shown. Here, participants made fast reaching movements while unexpectedly a 30-degree visuomotor transformation was introduced. During visuomotor adaptation, subjects received cerebellar, primary motor cortex (M1) or sham anodal transcranial direct current stimulation (tDCS), a noninvasive form of brain stimulation known to increase excitability. We found that cerebellar tDCS caused faster adaptation to the visuomotor transformation, as shown by a rapid reduction of movement errors. These findings were not present with similar modulation of visual cortex excitability. In contrast, tDCS over M1 did not affect adaptation, but resulted in a marked increase in retention of the newly learnt visuomotor transformation. These results show a clear dissociation in the processes of acquisition and retention during adaptive motor learning and demonstrate that the cerebellum and primary motor cortex have distinct functional roles. Furthermore, they show that is possible to enhance cerebellar function using tDCS.     

Relationship of Trauma Symptoms to Amygdala‐Based Functional Brain Changes in Adolescents

In this pilot study, amygdala connectivity related to trauma symptoms was explored using resting‐state functional magnetic resonance imaging (R‐fMRI) in 23 healthy adolescents ages 13–17 years with no psychiatric diagnoses. Adolescents completed a self‐report trauma symptom checklist and a R‐fMRI scan. We examined the relationship of trauma symptoms to resting‐state functional connectivity of the amygdala. Increasing self‐report of trauma symptoms by adolescents was associated with increasing functional connectivity with the right amygdala and a local limbic cluster and decreasing functional connectivity with the amygdala and a long‐range frontoparietal cluster to the left amygdala, which can be a hallmark of immaturity. These pilot findings in adolescents provide preliminary evidence that even mild trauma symptoms can be linked to the configuration of brain networks associated with the amygdala.     

The prefrontal cortex shows context-specific changes in effective connectivity to motor or visual cortex during the selection of action or colour

The role of the prefrontal cortex remains controversial. Neuroimaging studies support modality-specific and process-specific functions related to working memory and attention. Its role may also be defined by changes in its influence over other brain regions including sensory and motor cortex. We used functional magnetic imaging (fMRI) to study the free selection of actions and colours. Control conditions used externally specified actions and colours. The prefrontal cortex was activated during free selection, regardless of modality, in contrast to modality-specific activations outside prefrontal cortex. Structural equation modelling (SEM) of fMRI data was used to test the hypothesis that although the same regions of prefrontal cortex may be active in tasks within different domains, there is task-dependent effective connectivity between prefrontal cortex and non-prefrontal cortex. The SEM included high-order interactions between modality, selection and regional activity. There was greater coupling between prefrontal cortex and motor cortex during free selection and action tasks, and between prefrontal cortex and visual cortex during free selection of colours. The results suggest that the functions of the prefrontal cortex may be defined not only by selection-specific rather than modality-specific processes, but also by changing patterns of effective connectivity from prefrontal cortex to motor and sensory cortices.     

Signal-, set- and movement-related activity in the human brain: an event-related fMRI study

Electrophysiological studies on monkeys have been able to distinguish sensory and motor signals close in time by pseudorandomly delaying the cue that instructs the movement from the stimulus that triggers the movement. We have used a similar experimental design in functional magnetic resonance imaging (fMRI), scanning subjects while they performed a visuomotor conditional task with instructed delays. One of four shapes was presented briefly. Two shapes instructed the subjects to flex the index finger; the other two shapes coded the flexion of the middle finger. The subjects were told to perform the movement after a tone. We have exploited a novel use of event-related fMRI. By systematically varying the interval between the visual and acoustic stimuli, it has been possible to estimate the significance of the evoked haemodynamic response (EHR) to each of the stimuli, despite their temporal proximity in relation to the time constant of the EHR. Furthermore, by varying the phase between events and image acquisition, we have been able to achieve high temporal resolution while scanning the whole brain. We dissociated sensory and motor components of the sensorimotor transformations elicited by the task, and assessed sustained activity during the instructed delays. In calcarine and occipitotemporal cortex, the responses were exclusively associated with the visual instruction cues. In temporal auditory cortex and in primary motor cortex, they were exclusively associated with the auditory trigger stimulus. In ventral prefrontal cortex there were movement-related responses preceded by preparatory activity and by signal-related activity. Finally, responses associated with the instruction cue and with sustained activity during the delay period were observed in the dorsal premotor cortex and in the dorsal posterior parietal cortex. Where the association between a visual cue and the appropriate movement is arbitrary, the underlying visuomotor transformations are not achieved exclusively through frontoparietal interactions. Rather, these processes seem to rely on the ventral visual stream, the ventral prefrontal cortex and the anterior part of the dorsal premotor cortex.     

Understanding the neural computations of arbitrary visuomotor learning through fMRI and associative learning theory

Associative theory postulates that learning the consequences of our actions in a given context is represented in the brain as stimulus-response-outcome associations that evolve according to prediction-error signals (the discrepancy between the observed and predicted outcome). We tested the theory on brain functional magnetic resonance imaging data acquired from human participants learning arbitrary visuomotor associations. We developed a novel task that systematically manipulated learning and induced highly reproducible performances. This granted the validation of the model-based results and an in-depth analysis of the brain signals in representative single trials. Consistent with the Rescorla-Wagner model, prediction-error signals are computed in the human brain and selectively engage the ventral striatum. In addition, we found evidence of computations not formally predicted by the Rescorla-Wagner model. The dorsal fronto-parietal network, the dorsal striatum, and the ventrolateral prefrontal cortex are activated both on the incorrect and first correct trials and may reflect the processing of relevant visuomotor mappings during the early phases of learning. The left dorsolateral prefrontal cortex is selectively activated on the first correct outcome. The results provide quantitative evidence of the neural computations mediating arbitrary visuomotor learning and suggest new directions for future computational models.     

Functional connectivity of cortical networks involved in bimanual motor sequence learning

Motor skill learning requires the involvement and integration of several cortical and subcortical regions. In this study, we focus on how the functional connectivity of cortical networks changes with the acquisition of a novel motor skill. Using functional magnetic resonance imaging, we measured the localized blood oxygenation level-dependent (BOLD) signal in cortical regions while subjects performed a bimanual serial reaction time task under 2 conditions: 1) explicitly learning a novel sequence (NOVEL) and 2) playing a previously learned sequence (LEARNED). To investigate stages of learning, each condition was further divided into nonoverlapping early and late conditions. Functional connectivity was measured using a task-specific low-frequency coherence analysis of the data. We show that within the cortical motor network, the sensorimotor cortex, premotor cortex, and supplementary motor area have significantly greater inter- and intrahemispheric coupling during the early NOVEL condition compared with the late NOVEL condition. Additionally, we observed greater connectivity between frontal regions and cortical motor regions in the early versus late NOVEL contrast. No changes in functional connectivity were observed in the LEARNED condition. These results demonstrate that the functional connectivity of the cortical motor network is modulated with practice and suggest that early skill learning is mediated by enhanced interregional coupling.     

Olfactory learning modifies predisposition for long-term potentiation and long-term depression induction in the rat piriform (olfactory) cortex

Learning-related modifications in predisposition for long-term potentiation (LTP) and long-term depression (LTD) were studied in brain slices of the rat piriform cortex following olfactory learning. Rats were trained to discriminate between pairs of odors until they demonstrated rule learning. We have previously shown that such training is accompanied by enhanced neuronal excitability and increased synaptic transmission in the intrinsic synaptic pathway. Here we show that the susceptibility for further enhancing synaptic connectivity by inducing LTP in slices from trained rats is markedly reduced after training, compared with slices from pseudo-trained and naive rats. Accordingly, while 900 stimuli at 1 Hz did not induce LTD in slices from control rats, it induced significant LTD in slices from trained rats. Post-tetanic potentiation (PTP) was also reduced after training, indicating that synaptic release is enhanced after odor learning, as previously suggested. We suggest that learning-related cellular modifications and activity-dependent synaptic plasticity share a common mechanism in the primary olfactory cortex. Our data also support the prediction generated according to the sliding modification threshold theory that learning should be accompanied by reduced capability of inducing LTP and increased susceptibility for LTD induction.     

Intranasal Insulin Enhanced Resting-State Functional Connectivity of Hippocampal Regions in Type 2 Diabetes

Type 2 diabetes mellitus (T2DM) alters brain function and manifests as brain atrophy. Intranasal insulin has emerged as a promising intervention for treatment of cognitive impairment. We evaluated the acute effects of intranasal insulin on resting-state brain functional connectivity in older adults with T2DM. This proof-of-concept, randomized, double-blind, placebo-controlled study evaluated the effects of a single 40 IU dose of insulin or saline in 14 diabetic and 14 control subjects. Resting-state functional connectivity between the hippocampal region and default mode network (DMN) was quantified using functional MRI (fMRI) at 3Tesla. Following insulin administration, diabetic patients demonstrated increased resting-state connectivity between the hippocampal regions and the medial frontal cortex (MFC) as compared with placebo (cluster size: right, P = 0.03) and other DMN regions. On placebo, the diabetes group had lower connectivity between the hippocampal region and the MFC as compared with control subjects (cluster size: right, P = 0.02), but on insulin, MFC connectivity was similar to control subjects. Resting-state connectivity correlated with cognitive performance. A single dose of intranasal insulin increases resting-state functional connectivity between the hippocampal regions and multiple DMN regions in older adults with T2DM. Intranasal insulin administration may modify functional connectivity among brain regions regulating memory and complex cognitive behaviors.     

Resting-state brain networks in type 1 diabetic patients with and without microangiopathy and their relation to cognitive functions and disease variables

Cognitive functioning depends on intact brain networks that can be assessed with functional magnetic resonance imaging (fMRI) techniques. We hypothesized that cognitive decrements in type 1 diabetes mellitus (T1DM) are associated with alterations in resting-state neural connectivity and that these changes vary according to the degree of microangiopathy. T1DM patients with (MA(+): n = 49) and without (MA(-): n = 52) microangiopathy were compared with 48 healthy control subjects. All completed a neuropsychological assessment and resting-state fMRI. Networks were identified using multisubject independent component analysis; specific group differences within each network were analyzed using the dual-regression method, corrected for confounding factors and multiple comparisons. Relative to control subjects, MA(-) patients showed increased connectivity in networks involved in motor and visual processes, whereas MA(+) patients showed decreased connectivity in networks involving attention, working memory, auditory and language processing, and motor and visual processes. Better information-processing speed and general cognitive ability were related to increased degree of connectivity. T1DM is associated with a functional reorganization of neural networks that varies, dependent on the presence or absence of microangiopathy.     

Frontoparietal network involved in successful retrieval from episodic memory. Spatial and temporal analyses using fMRI and ERP

The neural basis for successful recognition of previously studied items, referred to as "retrieval success," has been investigated using either neuroimaging or brain potentials; however, few studies have used both modalities. Our study combined event-related functional magnetic resonance imaging (fMRI) and event-related potential (ERP) in separate groups of subjects. The neural responses were measured while the subjects performed an old/new recognition task with pictures that had been previously studied in either a deep- or shallow-encoding condition. The fMRI experiment showed that among the frontoparietal regions involved in retrieval success, the inferior frontal gyrus and intraparietal sulcus were crucial to conscious recollection because the activity of these regions was influenced by the depth of memory at encoding. The activity of the right parietal region in response to a repeated item was modulated by the repetition lag, indicating that this area would be critical to familiarity-based judgment. The results of structural equation modeling revealed that the functional connectivity among the regions in the left hemisphere was more significant than that in the right hemisphere. The results of the ERP experiment and independent component analysis paralleled those of the fMRI experiment and demonstrated that the repeated item produced an earlier peak than the hit item by approximately 50 ms.     

Motor "dexterity"?: Evidence that left hemisphere lateralization of motor circuit connectivity is associated with better motor performance in children

Motor control relies on well-established motor circuits, which are critical for typical child development. Although many imaging studies have examined task activation during motor performance, none have examined the relationship between functional intrinsic connectivity and motor ability. The current study investigated the relationship between resting state functional connectivity within the motor network and motor performance assessment outside of the scanner in 40 typically developing right-handed children. Better motor performance correlated with greater left-lateralized (mean left hemisphere-mean right hemisphere) motor circuit connectivity. Speed, rhythmicity, and control of movements were associated with connectivity within different individual region pairs: faster speed was associated with more left-lateralized putamen-thalamus connectivity, less overflow with more left-lateralized supplementary motor-primary motor connectivity, and less dysrhythmia with more left-lateralized supplementary motor-anterior cerebellar connectivity. These findings suggest that for right-handed children, superior motor development depends on the establishment of left-hemisphere dominance in intrinsic motor network connectivity.     

Endogenous testosterone modulates prefrontal-amygdala connectivity during social emotional behavior

It is clear that the steroid hormone testosterone plays an important role in the regulation of social emotional behavior, but it remains unknown which neural circuits mediate these hormonal influences in humans. We investigated the modulatory effects of endogenous testosterone on the control of social emotional behavior by applying functional magnetic resonance imaging while healthy male participants performed a social approach-avoidance task. This task operationalized social emotional behavior by having participants approach and avoid emotional faces by pulling and pushing a joystick, respectively. Affect-congruent trials mapped the automatic tendency to approach happy faces and avoid angry faces. Affect-incongruent trials required participants to override those automatic action tendencies and select the opposite response (approach-angry, avoid-happy). The social emotional control required by affect-incongruent responses resulted in longer reaction times (RTs) and increased activity at the border of the ventrolateral prefrontal cortex and frontal pole (VLPFC/FP). We show that endogenous testosterone modulates these cerebral congruency effects through 2 mechanisms. First, participants with lower testosterone levels generate larger VLPFC/FP responses during affect-incongruent trials. Second, during the same trials, endogenous testosterone modulates the effective connectivity between the VLPFC/FP and the amygdala. These results indicate that endogenous testosterone influences local prefrontal activity and interregional connectivity supporting the control of social emotional behavior.     

Influence of lesions detected by computed tomography on outcome and neuropsychological recovery after severe head injury

Outcome at 6 months after severe head injury was determined in 117 patients whose computed tomographic (CT) examinations demonstrated diffuse axonal injury (DAI), diffuse swelling (DS), or focal injuries. Neuropsychological sequelae were ascertained from two examinations in 30 of the conscious survivors within the 1st year after injury. Outcome differences varied with the type of CT lesion. DS and focal injuries resulted in more favorable (good recovery) outcomes. Mortality was higher after DAI. Neuropsychological outcome varied with the type of CT lesion and the function measured. Overall differences in memory and learning were revealed among the three CT lesion categories, whereas differences in intelligence and visuomotor functions were not significant. Levels of memory, learning, and visuomotor speed were higher after DS injuries, but improvement was less. Greater improvement of memory, learning, and visuomotor speed occurred after DAI. After focal injuries, visuomotor speed improved, but not recall and learning. The results suggest that the type of injury incurred differentially influences the outcome and the neuropsychological aftermath of severely head-injured adults.     

Evidence of left inferior frontal-premotor structural and functional connectivity deficits in adults who stutter

The neurophysiological basis for stuttering may involve deficits that affect dynamic interactions among neural structures supporting fluid speech processing. Here, we examined functional and structural connectivity within corticocortical and thalamocortical loops in adults who stutter. For functional connectivity, we placed seeds in the left and right inferior frontal Brodmann area 44 (BA44) and in the ventral lateral nucleus (VLN) of the thalamus. Subject-specific seeds were based on peak activation voxels captured during speech and nonspeech tasks using functional magnetic resonance imaging. Psychophysiological interaction (PPI) was used to find brain regions with heightened functional connectivity with these cortical and subcortical seeds during speech and nonspeech tasks. Probabilistic tractography was used to track white matter tracts in each hemisphere using the same seeds. Both PPI and tractrography supported connectivity deficits between the left BA44 and the left premotor regions, while connectivity among homologous right hemisphere structures was significantly increased in the stuttering group. No functional connectivity differences between BA44 and auditory regions were found between groups. The functional connectivity results derived from the VLN seeds were less definitive and were not supported by the tractography results. Our data provide strongest support for deficient left hemisphere inferior frontal to premotor connectivity as a neural correlate of stuttering.