Comparing apples and oranges: using reward-specific and reward-general subjective value representation in the brain

The ability of human subjects to choose between disparate kinds of rewards suggests that the neural circuits for valuing different reward types must converge. Economic theory suggests that these convergence points represent the subjective values (SVs) of different reward types on a common scale for comparison. To examine these hypotheses and to map the neural circuits for reward valuation we had food and water-deprived subjects make risky choices for money, food, and water both in and out of a brain scanner. We found that risk preferences across reward types were highly correlated; the level of risk aversion an individual showed when choosing among monetary lotteries predicted their risk aversion toward food and water. We also found that partially distinct neural networks represent the SVs of monetary and food rewards and that these distinct networks showed specific convergence points. The hypothalamic region mainly represented the SV for food, and the posterior cingulate cortex mainly represented the SV for money. In both the ventromedial prefrontal cortex (vmPFC) and striatum there was a common area representing the SV of both reward types, but only the vmPFC significantly represented the SVs of money and food on a common scale appropriate for choice in our data set. A correlation analysis demonstrated interactions across money and food valuation areas and the common areas in the vmPFC and striatum. This may suggest that partially distinct valuation networks for different reward types converge on a unified valuation network, which enables a direct comparison between different reward types and hence guides valuation and choice.     

Major depressive disorder is characterized by greater reward network activation to monetary than pleasant image rewards

Anhedonia, the loss of interest or pleasure in normally rewarding activities, is a hallmark feature of unipolar Major Depressive Disorder (MDD). A growing body of literature has identified frontostriatal dysfunction during reward anticipation and outcomes in MDD. However, no study to date has directly compared responses to different types of rewards such as pleasant images and monetary rewards in MDD. To investigate the neural responses to monetary and pleasant image rewards in MDD, a modified Monetary Incentive Delay task was used during functional magnetic resonance imaging to assess neural responses during anticipation and receipt of monetary and pleasant image rewards. Participants included nine adults with MDD and 13 affectively healthy controls. The MDD group showed lower activation than controls when anticipating monetary rewards in right orbitofrontal cortex and subcallosal cortex, and when anticipating pleasant image rewards in paracingulate and supplementary motor cortex. The MDD group had relatively greater activation in right putamen when anticipating monetary versus pleasant image rewards, relative to the control group. Results suggest reduced reward network activation in MDD when anticipating rewards, as well as relatively greater hypoactivation to pleasant image than monetary rewards.     

Impulsive personality and the ability to resist immediate reward: An fMRI study examining interindividual differences in the neural mechanisms underlying self-control

The ability to resist immediate rewards is crucial for lifetime success and individual well‐being. Using functional magnetic resonance imaging, we assessed the association between trait impulsivity and the neural underpinnings of the ability to control immediate reward desiring. Low and high extreme impulsivity groups were compared with regard to their behavioral performance and brain activation in situations, in which they had to forego immediate rewards with varying value to achieve a superordinate long‐term goal. We found that highly impulsive (HI) individuals, who successfully compensated for their lack in behavioral self‐control, engaged two complementary brain mechanisms when choosing actions in favor of a long‐term goal, but at the expense of an immediate reward. First, self‐controlled decisions led to a general attenuation of reward‐related activation in the nucleus accumbens, which was accompanied by an increased inverse connectivity with the anteroventral prefrontal cortex. Second, HI subjects controlled their desire for increasingly valuable, but suboptimal rewards through a linear reduction of activation in the ventromedial prefrontal cortex (VMPFC). This was achieved by an increased inverse coupling between the VMPFC and the ventral striatum. Importantly, the neural mechanisms observed in the HI group differed from those in extremely controlled individuals, despite similar behavioral performance. Collectively, these results suggest trait‐specific neural mechanisms that allow HI individuals to control their desire for immediate reward. Hum Brain Mapp, 2012. © 2011 Wiley Periodicals, Inc.     

Single-Dimensional Human Brain Signals for Two-Dimensional Economic Choice Options

Rewarding choice options typically contain multiple components, but neural signals in single brain voxels are scalar and primarily vary up or down. In a previous study, we had designed reward bundles that contained the same two milkshakes with independently set amounts; we had used psychophysics and rigorous economic concepts to estimate two-dimensional choice indifference curves (ICs) that represented revealed stochastic preferences for these bundles in a systematic, integrated manner. All bundles on the same ICs were equally revealed preferred (and thus had same utility, as inferred from choice indifference); bundles on higher ICs (higher utility) were preferred to bundles on lower ICs (lower utility). In the current study, we used the established behavior for testing with functional magnetic resonance imaging (fMRI). We now demonstrate neural responses in reward-related brain structures of human female and male participants, including striatum, midbrain, and medial orbitofrontal cortex (mid-OFC) that followed the characteristic pattern of ICs: similar responses along ICs (same utility despite different bundle composition), but monotonic change across ICs (different utility). Thus, these brain structures integrated multiple reward components into a scalar signal, well beyond the known subjective value coding of single-component rewards.SIGNIFICANCE STATEMENT Rewards have several components, like the taste and size of an apple, but it is unclear how each component contributes to the overall value of the reward. While choice indifference curves (ICs) of economic theory provide behavioral approaches to this question, it is unclear whether brain responses capture the preference and utility integrated from multiple components. We report activations in striatum, midbrain, and orbitofrontal cortex (OFC) that follow choice ICs representing behavioral preferences over and above variations of individual reward components. In addition, the concept-driven approach encourages future studies on natural, multicomponent rewards that are prone to irrational choice of normal and brain-damaged individuals.     

Effects of emotional preferences on value-based decision-making are mediated by mentalizing and not reward networks

Real-world decision-making often involves social considerations. Consequently, the social value of stimuli can induce preferences in choice behavior. However, it is unknown how financial and social values are integrated in the brain. Here, we investigated how smiling and angry face stimuli interacted with financial reward feedback in a stochastically rewarded decision-making task. Subjects reliably preferred the smiling faces despite equivalent reward feedback, demonstrating a socially driven bias. We fit a Bayesian reinforcement learning model to factor the effects of financial rewards and emotion preferences in individual subjects, and regressed model predictions on the trial-by-trial fMRI signal. Activity in the subcallosal cingulate and the ventral striatum, both involved in reward learning, correlated with financial reward feedback, whereas the differential contribution of social value activated dorsal temporo-parietal junction and dorsal anterior cingulate cortex, previously proposed as components of a mentalizing network. We conclude that the impact of social stimuli on value-based decision processes is mediated by effects in brain regions partially separable from classical reward circuitry.     

Valuation of uncertain and delayed rewards in primate prefrontal cortex

Humans and animals often must choose between rewards that differ in their qualities, magnitudes, immediacy, and likelihood, and must estimate these multiple reward parameters from their experience. However, the neural basis for such complex decision making is not well understood. To understand the role of the primate prefrontal cortex in determining the subjective value of delayed or uncertain reward, we examined the activity of individual prefrontal neurons during an inter-temporal choice task and a computer-simulated competitive game. Consistent with the findings from previous studies in humans and other animals, the monkey’s behaviors during inter-temporal choice were well accounted for by a hyperbolic discount function. In addition, the activity of many neurons in the lateral prefrontal cortex reflected the signals related to the magnitude and delay of the reward expected from a particular action, and often encoded the difference in temporally discounted values that predicted the animal’s choice. During a computerized matching pennies game, the animals approximated the optimal strategy, known as Nash equilibrium, using a reinforcement learning algorithm. We also found that many neurons in the lateral prefrontal cortex conveyed the signals related to the animal’s previous choices and their outcomes, suggesting that this cortical area might play an important role in forming associations between actions and their outcomes. These results show that the primate lateral prefrontal cortex plays a central role in estimating the values of alternative actions based on multiple sources of information.     

Decoding the formation of reward predictions across learning

The predicted reward of different behavioral options plays an important role in guiding decisions. Previous research has identified reward predictions in prefrontal and striatal brain regions. Moreover, it has been shown that the neural representation of a predicted reward is similar to the neural representation of the actual reward outcome. However, it has remained unknown how these representations emerge over the course of learning and how they relate to decision making. Here, we sought to investigate learning of predicted reward representations using functional magnetic resonance imaging and multivariate pattern classification. Using a pavlovian conditioning procedure, human subjects learned multiple novel cue-outcome associations in each scanning run. We demonstrate that across learning activity patterns in the orbitofrontal cortex, the dorsolateral prefrontal cortex (DLPFC), and the dorsal striatum, coding the value of predicted rewards become similar to the patterns coding the value of actual reward outcomes. Furthermore, we provide evidence that predicted reward representations in the striatum precede those in prefrontal regions and that representations in the DLPFC are linked to subsequent value-based choices. Our results show that different brain regions represent outcome predictions by eliciting the neural representation of the actual outcome. Furthermore, they suggest that reward predictions in the DLPFC are directly related to value-based choices.     

Brain reward system's alterations in response to food and monetary stimuli in overweight and obese individuals

The brain's reward system is crucial to understand obesity in modern society, as increased neural responsivity to reward can fuel the unhealthy food choices that are driving the growing obesity epidemic. Brain's reward system responsivity to food and monetary rewards in individuals with excessive weight (overweight and obese) versus normal weight controls, along with the relationship between this responsivity and body mass index (BMI) were tested. The sample comprised 21 adults with obesity (BMI > 30), 21 with overweight (BMI between 25 and 30), and 39 with normal weight (BMI < 25). Participants underwent a functional magnetic resonance imaging (fMRI) session while performing two tasks that involve the processing of food (Willing to Pay) and monetary rewards (Monetary Incentive Delay). Neural activations within the brain reward system were compared across the three groups. Curve fit analyses were conducted to establish the association between BMI and brain reward system's response. Individuals with obesity had greater food‐evoked responsivity in the dorsal and ventral striatum compared with overweight and normal weight groups. There was an inverted U‐shape association between BMI and monetary‐evoked responsivity in the ventral striatum, medial frontal cortex, and amygdala; that is, individuals with BMIs between 27 and 32 had greater responsivity to monetary stimuli. Obesity is associated with greater food‐evoked responsivity in the ventral and dorsal striatum, and overweight is associated with greater monetary‐evoked responsivity in the ventral striatum, the amygdala, and the medial frontal cortex. Findings suggest differential reactivity of the brain's reward system to food versus monetary rewards in obesity and overweight. Hum Brain Mapp 38:666–677, 2017. © 2016 Wiley Periodicals, Inc.     

A common currency for the computation of motivational values in the human striatum

Reward comparison in the brain is thought to be achieved through the use of a ‘common currency’, implying that reward value representations are computed on a unique scale in the same brain regions regardless of the reward type. Although such a mechanism has been identified in the ventro-medial prefrontal cortex and ventral striatum in the context of decision-making, it is less clear whether it similarly applies to non-choice situations. To answer this question, we scanned 38 participants with fMRI while they were presented with single cues predicting either monetary or erotic rewards, without the need to make a decision. The ventral striatum was the main brain structure to respond to both cues while showing increasing activity with increasing expected reward intensity. Most importantly, the relative response of the striatum to monetary vs erotic cues was correlated with the relative motivational value of these rewards as inferred from reaction times. Similar correlations were observed in a fronto-parietal network known to be involved in attentional focus and motor readiness. Together, our results suggest that striatal reward value signals not only obey to a common currency mechanism in the absence of choice but may also serve as an input to adjust motivated behaviour accordingly.     

Social and monetary reward learning engage overlapping neural substrates

Learning to make choices that yield rewarding outcomes requires the computation of three distinct signals: stimulus values that are used to guide choices at the time of decision making, experienced utility signals that are used to evaluate the outcomes of those decisions and prediction errors that are used to update the values assigned to stimuli during reward learning. Here we investigated whether monetary and social rewards involve overlapping neural substrates during these computations. Subjects engaged in two probabilistic reward learning tasks that were identical except that rewards were either social (pictures of smiling or angry people) or monetary (gaining or losing money). We found substantial overlap between the two types of rewards for all components of the learning process: a common area of ventromedial prefrontal cortex (vmPFC) correlated with stimulus value at the time of choice and another common area of vmPFC correlated with reward magnitude and common areas in the striatum correlated with prediction errors. Taken together, the findings support the hypothesis that shared anatomical substrates are involved in the computation of both monetary and social rewards.     

Neural correlates of frustration

Psychological considerations suggest that the omission of rewards in humans comprises two effects: first, an allocentric effect triggering learning and behavioural changes potentially processed by dopaminergic neurons according to the prediction error theory; second, an egocentric effect representing the individual's emotional reaction, commonly called frustration. We investigated this second effect in the context of omission of monetary reward with functional magnetic resonance imaging. As expected, the contrast omission relative to receipt of reward led to a decrease in ventral striatal activation consistent with prediction error theory. Increased activation for this contrast was found in areas previously related to emotional pain: the right anterior insula and the right ventral prefrontal cortex. We interpreted this as a neural correlate of the egocentric effect.     

Anticipating control over aversive stimuli is mediated by the medial prefrontal cortex: An fMRI study with healthy adults

The anticipation of control over aversive events in life is relevant for our mental health. Insights on the underlying neural mechanisms remain limited. We developed a new functional magnetic resonance imaging (fMRI) task that uses auditory stimuli to explore the neural correlates of (1) the anticipation of control over aversion and (2) the processing of aversion. In a sample of 25 healthy adults, we observed increased neural activation in the medial prefrontal cortex (ventromedial prefrontal cortex and rostral anterior cingulate cortex), other brain areas relevant for reward anticipation (ventral striatum, brainstem [ventral tegmental area], midcingulate cortex), and the posterior cingulate cortex when they anticipated control over aversion compared with anticipating no control (1). The processing of aversive sounds compared to neutral sounds (2) was associated with increased neural activation in the bilateral posterior insula. Our findings provide evidence for the important role of medial prefrontal regions in control anticipation and highlight the relevance of conceiving the neural mechanisms involved within a reward‐based framework.     

The neurobiology of intertemporal choice: insight from imaging and lesion studies

People are frequently faced with intertemporal choices, i.e., choices differing in the timing of their consequences, preferring smaller rewards available immediately over larger rewards delivered after a delay. The inability to forgo sooner gratification to favor delayed reward (e.g., impulsivity) has been related to several pathological conditions characterized by poor self-control, including drug addiction and obesity. Comparative and functional human studies have implicated a network of brain areas involved in intertemporal choice, including the medial portion of the orbitofrontal cortex (mOFC). Moreover, damage to this cortical area increases preference for immediate gratification in intertemporal decisions. Here, we review recent neuroscientific studies concerning intertemporal choice, suggesting that the mOFC contributes to preference for delayed rewards, either by computing the value of future outcomes (i.e., valuation), or by enabling people to imagine and represent future rewards and their consequences (e.g., prospection).     

Re-evaluating the role of the orbitofrontal cortex in reward and reinforcement

The orbitofrontal cortex and adjacent ventromedial prefrontal cortex carry reward representations and mediate flexible behaviour when circumstances change. Here we review how recent experiments in humans and macaques have confirmed the existence of a major difference between the functions of the ventromedial prefrontal cortex and adjacent medial orbitofrontal cortex (mOFC) on the one hand and the lateral orbitofrontal cortex (lOFC) on the other. These differences, however, may not be best accounted for in terms of specializations for reward and error/punishment processing as is commonly assumed. Instead we argue that both lesion and functional magnetic resonance imaging studies reveal that the lOFC is concerned with the assignment of credit for both reward and error outcomes to the choice of specific stimuli and with the linking of specific stimulus representations to representations of specific types of reward outcome. By contrast, we argue that the ventromedial prefrontal cortex/mOFC is concerned with evaluation, value-guided decision-making and maintenance of a choice over successive decisions. Despite the popular view that they cause perseveration of behaviour and inability to inhibit repetition of a previously made choice, we found that lesions in neither orbitofrontal subdivision caused perseveration. On the contrary, lesions in the lOFC made animals switch more rapidly between choices when they were finding it difficult to assign reward values to choices. Lesions in the mOFC caused animals to lose their normal predisposition to repeat previously successful choices, suggesting that the mOFC does not just mediate value comparison in choice but also facilitates maintenance of the same choice if it has been successful.     

The neural correlates of intertemporal decision-making: contributions of subjective value, stimulus type, and trait impulsivity

Making choices between payoffs available at different points in time reliably engages a decision-making brain circuit that includes medial prefrontal cortex (mPFC), posterior cingulate cortex (PCC), and ventral striatum (VS). Previous neuroimaging studies produced differing accounts of the functions of these regions, including that these regions: (1) are sensitive to the value of rewards discounted by a function of delay ('subjective value'); (2) are differentially sensitive to the availability of an immediate reward; and (3) are implicated in impulsive decision-making. In this event-related fMRI study of 20 volunteers, these hypotheses were investigated simultaneously using a delay discounting task in which magnitude of rewards and stimulus type, i.e., the presence or absence of an immediate option, were independently varied, and in which participants' trait impulsivity was assessed with the Barratt Impulsiveness Scale. Results showed that mPFC, PCC, and VS are sensitive to the subjective value of rewards, whereas mPFC and PCC, but not VS, are sensitive to the presence of an immediate reward in the choice option. Moderation by individual differences in trait impulsivity was specific to the mPFC. Conjunction analysis showed significant overlap in mPFC and PCC for the main effects of subjective value and stimulus type, indicating these regions may serve multiple distinct roles during intertemporal decision-making. These findings significantly advance our understanding of the specificity and overlap of functions subserved by different regions involved in intertemporal decision-making, and help to reconcile conflicting accounts in the literature.     

Adaptive neural reward processing during anticipation and receipt of monetary rewards in mindfulness meditators

Reward seeking is ubiquitous and adaptive in humans. But excessive reward seeking behavior, such as chasing monetary rewards, may lead to diminished subjective well-being. This study examined whether individuals trained in mindfulness meditation show neural evidence of lower susceptibility to monetary rewards. Seventy-eight participants (34 meditators, 44 matched controls) completed the monetary incentive delay task while undergoing functional magnetic resonance imaging. The groups performed equally on the task, but meditators showed lower neural activations in the caudate nucleus during reward anticipation, and elevated bilateral posterior insula activation during reward anticipation. Meditators also evidenced reduced activations in the ventromedial prefrontal cortex during reward receipt compared with controls. Connectivity parameters between the right caudate and bilateral anterior insula were attenuated in meditators during incentive anticipation. In summary, brain regions involved in reward processing—both during reward anticipation and receipt of reward—responded differently in mindfulness meditators than in nonmeditators, indicating that the former are less susceptible to monetary incentives.     

The neuroeconomics of nicotine dependence: a preliminary functional magnetic resonance imaging study of delay discounting of monetary and cigarette rewards in smokers

Neuroeconomics integrates behavioral economics and cognitive neuroscience to understand the neurobiological basis for normative and maladaptive decision making. Delay discounting is a behavioral economic index of impulsivity that reflects capacity to delay gratification and has been consistently associated with nicotine dependence. This preliminary study used functional magnetic resonance imaging to examine delay discounting for money and cigarette rewards in 13 nicotine dependent adults. Significant differences between preferences for smaller immediate rewards and larger delayed rewards were evident in a number of regions of interest (ROIs), including the medial prefrontal cortex, anterior insular cortex, middle temporal gyrus, middle frontal gyrus, and cingulate gyrus. Significant differences between money and cigarette rewards were generally lateralized, with cigarette choices associated with left hemisphere activation and money choices associated with right hemisphere activation. Specific ROI differences included the posterior parietal cortex, medial and middle frontal gyrus, ventral striatum, temporoparietal cortex, and angular gyrus. Impulsivity as measured by behavioral choices was significantly associated with both individual ROIs and a combined ROI model. These findings provide initial evidence in support of applying a neuroeconomic approach to understanding nicotine dependence.     

Regionally distinct processing of rewards and punishments by the primate ventromedial prefrontal cortex

The ventromedial prefrontal cortex (vmPFC) is thought to be related to emotional experience and to the processing of stimulus and action values. However, little is known about how single vmPFC neurons process the prediction and reception of rewards and punishments. We recorded from monkey vmPFC neurons in an experimental situation with alternating blocks, one in which rewards were delivered and one in which punishments were delivered. Many vmPFC neurons changed their activity between blocks. Importantly, neurons in ventral vmPFC were persistently more active in the appetitive "reward" block, whereas neurons in dorsal vmPFC were persistently more active in the aversive "punishment" block. Furthermore, within ventral vmPFC, posterior neurons phasically encoded probability of reward, whereas anterior neurons tonically encoded possibility of reward. We found multiple distinct nonlinear valuation mechanisms within the primate prefrontal cortex. Our findings suggest that different subregions of vmPFC contribute differentially to the processing of valence. By conveying such multidimensional and nonlinear signals, the vmPFC may enable flexible control of decisions and emotions to adapt to complex environments.     

Ventrolateral prefrontal cortex is required for performance of a strategy implementation task but not reinforcer devaluation effects in rhesus monkeys

The ability to apply behavioral strategies to obtain rewards efficiently and make choices based on changes in the value of rewards is fundamental to the adaptive control of behavior. The extent to which different regions of the prefrontal cortex are required for specific kinds of decisions is not well understood. We tested rhesus monkeys with bilateral ablations of the ventrolateral prefrontal cortex on tasks that required the use of behavioral strategies to optimize the rate with which rewards were accumulated, or to modify choice behavior in response to changes in the value of particular rewards. Monkeys with ventrolateral prefrontal lesions were impaired in performing the strategy-based task, but not on value-based decision-making. In contrast, orbital prefrontal ablations produced the opposite impairments in the same tasks. These findings support the conclusion that independent neural systems within the prefrontal cortex are necessary for control of choice behavior based on strategies or on stimulus value.     

Risk‐preference differentiates orbitofrontal cortex responses to freely chosen reward outcomes

To successfully evaluate potential courses of action and choose the most favorable, we must consider the outcomes that may result. Many choices involve risk, our assessment of which may lead us to success or failure in matters financial, legal or health‐related. The orbitofrontal cortex (OFC) has been implicated as critical for evaluating choices based on risk. To measure how outcomes of risky decisions are represented in the OFC, we recorded the electrophysiological activity of single neurons while rats made behavioral responses to obtain rewards under conditions of either certainty or risk. Rats exhibited different risk‐preferences when given the opportunity to choose. In risk‐preferring rats, OFC responses were enhanced following the delivery of large rewards obtained under risk compared with smaller, certain rewards and reward omission. However, in risk‐neutral rats, neurons showed similarly enhanced responses to both large rewards obtained under risk and smaller, certain rewards compared with reward omission. Thus, the responses of OFC neurons reflected the subjective evaluation of outcomes in individuals with different risk‐preferences. Such enhanced neural responding to risky rewards may serve to bias individuals towards risk‐preference in decision‐making.