Individual differences in sensitivity to reward and punishment and neural activity during reward and avoidance learning

In this functional neuroimaging study, we investigated neural activations during the process of learning to gain monetary rewards and to avoid monetary loss, and how these activations are modulated by individual differences in reward and punishment sensitivity. Healthy young volunteers performed a reinforcement learning task where they chose one of two fractal stimuli associated with monetary gain (reward trials) or avoidance of monetary loss (avoidance trials). Trait sensitivity to reward and punishment was assessed using the behavioral inhibition/activation scales (BIS/BAS). Functional neuroimaging results showed activation of the striatum during the anticipation and reception periods of reward trials. During avoidance trials, activation of the dorsal striatum and prefrontal regions was found. As expected, individual differences in reward sensitivity were positively associated with activation in the left and right ventral striatum during reward reception. Individual differences in sensitivity to punishment were negatively associated with activation in the left dorsal striatum during avoidance anticipation and also with activation in the right lateral orbitofrontal cortex during receiving monetary loss. These results suggest that learning to attain reward and learning to avoid loss are dependent on separable sets of neural regions whose activity is modulated by trait sensitivity to reward or punishment.     

Cognitive strategies regulate fictive,but not reward prediction error signals in a sequential investment task

Computational models of reward processing suggest that foregone or fictive outcomes serve as important information sources for learning and augment those generated by experienced rewards (e.g. reward prediction errors). An outstanding question is how these learning signals interact with top‐down cognitive influences, such as cognitive reappraisal strategies. Using a sequential investment task and functional magnetic resonance imaging, we show that the reappraisal strategy selectively attenuates the influence of fictive, but not reward prediction error signals on investment behavior; such behavioral effect is accompanied by changes in neural activity and connectivity in the anterior insular cortex, a brain region thought to integrate subjective feelings with high‐order cognition. Furthermore, individuals differ in the extent to which their behaviors are driven by fictive errors versus reward prediction errors, and the reappraisal strategy interacts with such individual differences; a finding also accompanied by distinct underlying neural mechanisms. These findings suggest that the variable interaction of cognitive strategies with two important classes of computational learning signals (fictive, reward prediction error) represent one contributing substrate for the variable capacity of individuals to control their behavior based on foregone rewards. These findings also expose important possibilities for understanding the lack of control in addiction based on possibly foregone rewarding outcomes. Hum Brain Mapp 35:3738–3749, 2014. © 2013 The Authors. Human Brain Mapping Published by Wiley Periodicals, Inc.     

Mapping adolescent reward anticipation,receipt, and prediction error during the monetary incentive delay task

The functional neuroanatomy and connectivity of reward processing in adults are well documented, with relatively less research on adolescents, a notable gap given this developmental period's association with altered reward sensitivity. Here, a large sample (n = 1,510) of adolescents performed the monetary incentive delay (MID) task during functional magnetic resonance imaging. Probabilistic maps identified brain regions that were reliably responsive to reward anticipation and receipt, and to prediction errors derived from a computational model. Psychophysiological interactions analyses were used to examine functional connections throughout reward processing. Bilateral ventral striatum, pallidum, insula, thalamus, hippocampus, cingulate cortex, midbrain, motor area, and occipital areas were reliably activated during reward anticipation. Bilateral ventromedial prefrontal cortex and bilateral thalamus exhibited positive and negative activation, respectively, during reward receipt. Bilateral ventral striatum was reliably active following prediction errors. Previously, individual differences in the personality trait of sensation seeking were shown to be related to individual differences in sensitivity to reward outcome. Here, we found that sensation seeking scores were negatively correlated with right inferior frontal gyrus activity following reward prediction errors estimated using a computational model. Psychophysiological interactions demonstrated widespread cortical and subcortical connectivity during reward processing, including connectivity between reward‐related regions with motor areas and the salience network. Males had more activation in left putamen, right precuneus, and middle temporal gyrus during reward anticipation. In summary, we found that, in adolescents, different reward processing stages during the MID task were robustly associated with distinctive patterns of activation and of connectivity.     

Separate brain regions code for salience vs. valence during reward prediction in humans

Predicting rewards and avoiding aversive conditions is essential for survival. Recent studies using computational models of reward prediction implicate the ventral striatum in appetitive rewards. Whether the same system mediates an organism's response to aversive conditions is unclear. We examined the question using fMRI blood oxygen level-dependent measurements while healthy volunteers were conditioned using appetitive and aversive stimuli. The temporal difference learning algorithm was used to estimate reward prediction error. Activations in the ventral striatum were robustly correlated with prediction error, regardless of the valence of the stimuli, suggesting that the ventral striatum processes salience prediction error. In contrast, the orbitofrontal cortex and anterior insula coded for the differential valence of appetitive/aversive stimuli. Given its location at the interface of limbic and motor regions, the ventral striatum may be critical in learning about motivationally salient stimuli, regardless of valence, and using that information to bias selection of actions.     

Effects of emotional preferences on value-based decision-making are mediated by mentalizing and not reward networks

Real-world decision-making often involves social considerations. Consequently, the social value of stimuli can induce preferences in choice behavior. However, it is unknown how financial and social values are integrated in the brain. Here, we investigated how smiling and angry face stimuli interacted with financial reward feedback in a stochastically rewarded decision-making task. Subjects reliably preferred the smiling faces despite equivalent reward feedback, demonstrating a socially driven bias. We fit a Bayesian reinforcement learning model to factor the effects of financial rewards and emotion preferences in individual subjects, and regressed model predictions on the trial-by-trial fMRI signal. Activity in the subcallosal cingulate and the ventral striatum, both involved in reward learning, correlated with financial reward feedback, whereas the differential contribution of social value activated dorsal temporo-parietal junction and dorsal anterior cingulate cortex, previously proposed as components of a mentalizing network. We conclude that the impact of social stimuli on value-based decision processes is mediated by effects in brain regions partially separable from classical reward circuitry.     

Signed Reward Prediction Errors in the Ventral Striatum Drive Episodic Memory

Recent behavioral evidence implicates reward prediction errors (RPEs) as a key factor in the acquisition of episodic memory. Yet, important neural predictions related to the role of RPEs in episodic memory acquisition remain to be tested. Humans (both sexes) performed a novel variable-choice task where we experimentally manipulated RPEs and found support for key neural predictions with fMRI. Our results show that in line with previous behavioral observations, episodic memory accuracy increases with the magnitude of signed (i.e., better/worse-than-expected) RPEs (SRPEs). Neurally, we observe that SRPEs are encoded in the ventral striatum (VS). Crucially, we demonstrate through mediation analysis that activation in the VS mediates the experimental manipulation of SRPEs on episodic memory accuracy. In particular, SRPE-based responses in the VS (during learning) predict the strength of subsequent episodic memory (during recollection). Furthermore, functional connectivity between task-relevant processing areas (i.e., face-selective areas) and hippocampus and ventral striatum increased as a function of RPE value (during learning), suggesting a central role of these areas in episodic memory formation. Our results consolidate reinforcement learning theory and striatal RPEs as key factors subtending the formation of episodic memory.SIGNIFICANCE STATEMENT Recent behavioral research has shown that reward prediction errors (RPEs), a key concept of reinforcement learning theory, are crucial to the formation of episodic memories. In this study, we reveal the neural underpinnings of this process. Using fMRI, we show that signed RPEs (SRPEs) are encoded in the ventral striatum (VS), and crucially, that SRPE VS activity is responsible for the subsequent recollection accuracy of one-shot learned episodic memory associations.     

Dorsal striatal-midbrain connectivity in humans predicts how reinforcements are used to guide decisions

It has been suggested that the target areas of dopaminergic midbrain neurons, the dorsal (DS) and ventral striatum (VS), are differently involved in reinforcement learning especially as actor and critic. Whereas the critic learns to predict rewards, the actor maintains action values to guide future decisions. The different midbrain connections to the DS and the VS seem to play a critical role in this functional distinction. Here, subjects performed a dynamic, reward-based decision-making task during fMRI acquisition. A computational model of reinforcement learning was used to estimate the different effects of positive and negative reinforcements on future decisions for each subject individually. We found that activity in both the DS and the VS correlated with reward prediction errors. Using functional connectivity, we show that the DS and the VS are differentially connected to different midbrain regions (possibly corresponding to the substantia nigra [SN] and the ventral tegmental area [VTA], respectively). However, only functional connectivity between the DS and the putative SN predicted the impact of different reinforcement types on future behavior. These results suggest that connections between the putative SN and the DS are critical for modulating action values in the DS according to both positive and negative reinforcements to guide future decision making.     

Neural processing of reward in adolescent rodents

Immaturities in adolescent reward processing are thought to contribute to poor decision making and increased susceptibility to develop addictive and psychiatric disorders. Very little is known; however, about how the adolescent brain processes reward. The current mechanistic theories of reward processing are derived from adult models. Here we review recent research focused on understanding of how the adolescent brain responds to rewards and reward-associated events. A critical aspect of this work is that age-related differences are evident in neuronal processing of reward-related events across multiple brain regions even when adolescent rats demonstrate behavior similar to adults. These include differences in reward processing between adolescent and adult rats in orbitofrontal cortex and dorsal striatum. Surprisingly, minimal age related differences are observed in ventral striatum, which has been a focal point of developmental studies. We go on to discuss the implications of these differences for behavioral traits affected in adolescence, such as impulsivity, risk-taking, and behavioral flexibility. Collectively, this work suggests that reward-evoked neural activity differs as a function of age and that regions such as the dorsal striatum that are not traditionally associated with affective processing in adults may be critical for reward processing and psychiatric vulnerability in adolescents.     

Dissociation of BOLD responses to reward prediction errors and reward receipt by a model comparison

The representation of reward anticipation and reward prediction errors is the basis for reward-associated learning. The representation of whether or not a reward occurred (reward receipt) is important for decision making. Recent studies suggest that, while reward anticipation and reward prediction errors are encoded in the midbrain and the ventral striatum, reward receipts are encoded in the medial orbitofrontal cortex. In order to substantiate this functional specialization we analyzed data from an fMRI study in which 59 subjects completed two simple monetary reward paradigms. Because reward receipts and reward prediction errors were correlated, a statistical model comparison was applied separating the effects of the two. Reward prediction error fitted BOLD responses significantly better than reward receipt in the midbrain and the ventral striatum. Conversely, reward receipt fitted BOLD responses better in the orbitofrontal cortex. Activation related to reward anticipation was found in the orbitofrontal cortex. The results confirm a functional specialization of behaviorally important aspects of reward processing within the mesolimbic dopaminergic system.     

Social and monetary reward learning engage overlapping neural substrates

Learning to make choices that yield rewarding outcomes requires the computation of three distinct signals: stimulus values that are used to guide choices at the time of decision making, experienced utility signals that are used to evaluate the outcomes of those decisions and prediction errors that are used to update the values assigned to stimuli during reward learning. Here we investigated whether monetary and social rewards involve overlapping neural substrates during these computations. Subjects engaged in two probabilistic reward learning tasks that were identical except that rewards were either social (pictures of smiling or angry people) or monetary (gaining or losing money). We found substantial overlap between the two types of rewards for all components of the learning process: a common area of ventromedial prefrontal cortex (vmPFC) correlated with stimulus value at the time of choice and another common area of vmPFC correlated with reward magnitude and common areas in the striatum correlated with prediction errors. Taken together, the findings support the hypothesis that shared anatomical substrates are involved in the computation of both monetary and social rewards.     

Frontostriatal response to set switching is moderated by reward sensitivity

The reinforcement sensitivity theory (RST) relates individual differences in reward sensitivity to the activation of the behavioral approach system (BAS). Dopamine-related brain structures have been repeatedly associated with reward processing, but also with cognitive processes such as task switching. In the present study, we examined the association between reward sensitivity and the event-related fMRI BOLD response with set switching in 31 males. As expected, the right inferior frontal cortex (rIFG) and the striatum (i.e. the left putamen) were involved in set-switching activity for the overall sample. Interindividual differences in Gray's reward sensitivity were related to stronger activity in the rIFG and the ventral striatum. Thus, trait reward sensitivity contributed to the modulation of brain responsiveness in set-switching tasks. Having considered previous research, we propose that higher BAS activity is associated with a stronger reward to process a better implementation of goal-directed tasks and the diminished processing of secondary cues.     

Individual associations of adolescent alcohol use disorder versus cannabis use disorder symptoms in neural prediction error signaling and the response to novelty

Two of the most commonly used illegal substances by adolescents are alcohol and cannabis. Alcohol use disorder (AUD) and cannabis use disorder (CUD) are associated with poorer decision-making in adolescents. In adolescents, level of AUD symptomatology has been negatively associated with striatal reward responsivity. However, little work has explored the relationship with striatal reward prediction error (RPE) representation and the extent to which any augmentation of RPE by novel stimuli is impacted. One-hundred fifty-one adolescents participated in the Novelty Task while undergoing functional magnetic resonance imaging (fMRI). In this task, participants learn to choose novel or non-novel stimuli to gain monetary reward. Level of AUD symptomatology was negatively associated with both optimal decision-making and BOLD response modulation by RPE within striatum and regions of prefrontal cortex. The neural alterations in RPE representation were particularly pronounced when participants were exploring novel stimuli. Level of CUD symptomatology moderated the relationship between novelty propensity and RPE representation within inferior parietal lobule and dorsomedial prefrontal cortex. These data expand on an emerging literature investigating individual associations of AUD symptomatology levels versus CUD symptomatology levels and RPE representation during reinforcement processing and provide insight on the role of neuro-computational processes underlying reinforcement learning/decision-making in adolescents.     

Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning.

The brain's most difficult computation in decision-making learning is searching for essential information related to rewards among vast multimodal inputs and then integrating it into beneficial behaviors. Contextual cues consisting of limbic, cognitive, visual, auditory, somatosensory, and motor signals need to be associated with both rewards and actions by utilizing an internal representation such as reward prediction and reward prediction error. Previous studies have suggested that a suitable brain structure for such integration is the neural circuitry associated with multiple cortico-striatal loops. However, computational exploration still remains into how the information in and around these multiple closed loops can be shared and transferred. Here, we propose a "heterarchical reinforcement learning" model, where reward prediction made by more limbic and cognitive loops is propagated to motor loops by spiral projections between the striatum and substantia nigra, assisted by cortical projections to the pedunculopontine tegmental nucleus, which sends excitatory input to the substantia nigra. The model makes several fMRI-testable predictions of brain activity during stimulus-action-reward association learning. The caudate nucleus and the cognitive cortical areas are correlated with reward prediction error, while the putamen and motor-related areas are correlated with stimulus-action-dependent reward prediction. Furthermore, a heterogeneous activity pattern within the striatum is predicted depending on learning difficulty, i.e., the anterior medial caudate nucleus will be correlated more with reward prediction error when learning becomes difficult, while the posterior putamen will be correlated more with stimulus-action-dependent reward prediction in easy learning. Our fMRI results revealed that different cortico-striatal loops are operating, as suggested by the proposed model.     

Brain mechanism of reward prediction under predictable and unpredictable environmental dynamics.

In learning goal-directed behaviors, an agent has to consider not only the reward given at each state but also the consequences of dynamic state transitions associated with action selection. To understand brain mechanisms for action learning under predictable and unpredictable environmental dynamics, we measured brain activities by functional magnetic resonance imaging (fMRI) during a Markov decision task with predictable and unpredictable state transitions. Whereas the striatum and orbitofrontal cortex (OFC) were significantly activated both under predictable and unpredictable state transition rules, the dorsolateral prefrontal cortex (DLPFC) was more strongly activated under predictable than under unpredictable state transition rules. We then modelled subjects' choice behaviours using a reinforcement learning model and a Bayesian estimation framework and found that the subjects took larger temporal discount factors under predictable state transition rules. Model-based analysis of fMRI data revealed different engagement of striatum in reward prediction under different state transition dynamics. The ventral striatum was involved in reward prediction under both unpredictable and predictable state transition rules, although the dorsal striatum was dominantly involved in reward prediction under predictable rules. These results suggest different learning systems in the cortico-striatum loops depending on the dynamics of the environment: the OFC-ventral striatum loop is involved in action learning based on the present state, while the DLPFC-dorsal striatum loop is involved in action learning based on predictable future states.     

Neural mechanisms of reward and loss processing in a low-income sample of at-risk adolescents

Adolescence is a time of engagement in risky, reward-driven behaviors, with concurrent developmental changes within reward-related neural systems. As previous research has recruited mostly higher socioeconomic, European and European American participants, therefore limiting generalizability to the US population, especially for populations of color or low-income populations. The current study provided one of the first opportunities to examine the neural correlates of reward and loss functioning in a population-based sample of adolescents at increased risk for poverty-related adversities. The study investigated neural reward and loss processing and whether age, pubertal status and the social constructs of gender and race predicted individual differences in reward- and loss-related brain function. One hundred and twenty-eight primarily low-income adolescents (mean age: 15.9 years, 75% African American) from urban environments completed a modified monetary incentive delay task during functional magnetic resonance imaging (fMRI). Consistent with the previous research, reward and loss anticipation recruited similar motivational circuitry including striatal, insular, thalamic and supplementary motor areas. Race and gender were not associated with reward- or loss-related neural reactivity. Age and pubertal development were associated with differences in neural reactivity to reward and loss, suggesting that older and more mature adolescents had increased activity in sensory and motivational circuits, but decreased activity in regions responsible for error detection and behavior modification.     

A reward prediction error for charitable donations reveals outcome orientation of donators

The motives underlying prosocial behavior, like charitable donations, can be related either to actions or to outcomes. To address the neural basis of outcome orientation in charitable giving, we asked 33 subjects to make choices affecting their own payoffs and payoffs to a charity organization, while being scanned by functional magnetic resonance imaging (fMRI). We experimentally induced a reward prediction error (RPE) by subsequently discarding some of the chosen outcomes. Co-localized to a nucleus accumbens BOLD signal corresponding to the RPE for the subject''s own payoff, we observed an equivalent RPE signal for the charity''s payoff in those subjects who were willing to donate. This unique demonstration of a neuronal RPE signal for outcomes exclusively affecting unrelated others indicates common brain processes during outcome evaluation for selfish, individual and nonselfish, social rewards and strongly suggests the effectiveness of outcome-oriented motives in charitable giving.     

Attenuated Directed Exploration during Reinforcement Learning in Gambling Disorder

Gambling disorder (GD) is a behavioral addiction associated with impairments in value-based decision-making and behavioral flexibility and might be linked to changes in the dopamine system. Maximizing long-term rewards requires a flexible trade-off between the exploitation of known options and the exploration of novel options for information gain. This exploration-exploitation trade-off is thought to depend on dopamine neurotransmission. We hypothesized that human gamblers would show a reduction in directed (uncertainty-based) exploration, accompanied by changes in brain activity in a fronto-parietal exploration-related network. Twenty-three frequent, non-treatment seeking gamblers and twenty-three healthy matched controls (all male) performed a four-armed bandit task during functional magnetic resonance imaging (fMRI). Computational modeling using hierarchical Bayesian parameter estimation revealed signatures of directed exploration, random exploration, and perseveration in both groups. Gamblers showed a reduction in directed exploration, whereas random exploration and perseveration were similar between groups. Neuroimaging revealed no evidence for group differences in neural representations of basic task variables (expected value, prediction errors). Our hypothesis of reduced frontal pole (FP) recruitment in gamblers was not supported. Exploratory analyses showed that during directed exploration, gamblers showed reduced parietal cortex and substantia-nigra/ventral-tegmental-area activity. Cross-validated classification analyses revealed that connectivity in an exploration-related network was predictive of group status, suggesting that connectivity patterns might be more predictive of problem gambling than univariate effects. Findings reveal specific reductions of strategic exploration in gamblers that might be linked to altered processing in a fronto-parietal network and/or changes in dopamine neurotransmission implicated in GD.SIGNIFICANCE STATEMENT Wiehler et al. (2021) report that gamblers rely less on the strategic exploration of unknown, but potentially better rewards during reward learning. This is reflected in a related network of brain activity. Parameters of this network can be used to predict the presence of problem gambling behavior in participants.     

Anticipation of monetary and social reward differently activates mesolimbic brain structures in men and women

Motivation for goal-directed behaviour largely depends on the expected value of the anticipated reward. The aim of the present study was to examine how different levels of reward value are coded in the brain for two common forms of human reward: money and social approval. To account for gender differences 16 male and 16 female participants performed an incentive delay task expecting to win either money or positive social feedback. fMRI recording during the anticipation phase revealed proportional activation of neural structures constituting the human reward system for increasing levels of reward, independent of incentive type. However, in men activation in the prospect of monetary rewards encompassed a wide network of mesolimbic brain regions compared to only limited activation for social rewards. In contrast, in women, anticipation of either incentive type activated identical brain regions. Our findings represent an important step towards a better understanding of motivated behaviour by taking into account individual differences in reward valuation.     

Functional dynamics of dopamine synthesis during monetary reward and punishment processing

The assessment of dopamine release with the PET competition model is thoroughly validated but entails disadvantages for the investigation of cognitive processes. We introduce a novel approach incorporating 6-[¹⁸F]FDOPA uptake as index of the dynamic regulation of dopamine synthesis enzymes by neuronal firing. The feasibility of this approach is demonstrated by assessing widely described sex differences in dopamine neurotransmission. Reward processing was behaviorally investigated in 36 healthy participants, of whom 16 completed fPET and fMRI during the monetary incentive delay task. A single 50 min fPET acquisition with 6-[¹⁸F]FDOPA served to quantify task-specific changes in dopamine synthesis. In men monetary gain induced stronger increases in ventral striatum dopamine synthesis than loss. Interestingly, the opposite effect was discovered in women. These changes were further associated with reward (men) and punishment sensitivity (women). As expected, fMRI showed robust task-specific neuronal activation but no sex difference. Our findings provide a neurobiological basis for known behavioral sex differences in reward and punishment processing, with important implications in psychiatric disorders showing sex-specific prevalence, altered reward processing and dopamine signaling. The high temporal resolution and magnitude of task-specific changes make fPET a promising tool to investigate functional neurotransmitter dynamics during cognitive processing and in brain disorders.     

A common mechanism for adaptive scaling of reward and novelty

Declarative memory is remarkably adaptive in the way it maintains sensitivity to relative novelty in both unknown and highly familiar environments. However, the neural mechanisms underlying this contextual adaptation are poorly understood. On the basis of emerging links between novelty processing and reinforcement learning mechanisms, we hypothesized that responses to novelty will be adaptively scaled according to expected contextual probabilities of new and familiar events, in the same way that responses to prediction errors for rewards are scaled according to their expected range. Using functional magnetic resonance imaging in humans, we show that the influence of novelty and reward on memory formation in an incidental memory task is adaptively scaled and furthermore that the BOLD signal in orbital prefrontal and medial temporal cortices exhibits concomitant scaled adaptive coding. These findings demonstrate a new mechanism for adjusting gain and sensitivity in declarative memory in accordance with contextual probabilities and expectancies of future events. Hum Brain Mapp, 2010. © 2010 Wiley‐Liss, Inc.