Individual differences and the neural representations of reward expectation and reward prediction error

Reward expectation and reward prediction errors are thoughtto be critical for dynamic adjustments in decision-making andreward-seeking behavior, but little is known about their representationin the brain during uncertainty and risk-taking. Furthermore,little is known about what role individual differences mightplay in such reinforcement processes. In this study, it is shownbehavioral and neural responses during a decision-making taskcan be characterized by a computational reinforcement learningmodel and that individual differences in learning parametersin the model are critical for elucidating these processes. Inthe fMRI experiment, subjects chose between high- and low-riskrewards. A computational reinforcement learning model computedexpected values and prediction errors that each subject mightexperience on each trial. These outputs predicted subjects’trial-to-trial choice strategies and neural activity in severallimbic and prefrontal regions during the task. Individual differencesin estimated reinforcement learning parameters proved criticalfor characterizing these processes, because models that incorporatedindividual learning parameters explained significantly morevariance in the fMRI data than did a model using fixed learningparameters. These findings suggest that the brain engages areinforcement learning process during risk-taking and that individualdifferences play a crucial role in modeling this process.     

Abnormal prediction error processing in schizophrenia and depression

To make adaptive decisions under uncertainty, individuals need to actively monitor the discrepancy between expected outcomes and actual outcomes, known as prediction errors. Reward‐based learning deficits have been shown in both depression and schizophrenia patients. For this study, we compiled studies that investigated prediction error processing in depression and schizophrenia patients and performed a series of meta‐analyses. In both groups, positive t‐maps of prediction error tend to yield striatum activity across studies. The analysis of negative t‐maps of prediction error revealed two large clusters within the right superior and inferior frontal lobes in schizophrenia and the medial prefrontal cortex and bilateral insula in depression. The concordant posterior cingulate activity was observed in both patient groups, more prominent in the depression group and absent in the healthy control group. These findings suggest a possible role in dopamine‐rich areas associated with the encoding of prediction errors in depression and schizophrenia.     

Neural mechanisms of reward and loss processing in a low-income sample of at-risk adolescents

Adolescence is a time of engagement in risky, reward-driven behaviors, with concurrent developmental changes within reward-related neural systems. As previous research has recruited mostly higher socioeconomic, European and European American participants, therefore limiting generalizability to the US population, especially for populations of color or low-income populations. The current study provided one of the first opportunities to examine the neural correlates of reward and loss functioning in a population-based sample of adolescents at increased risk for poverty-related adversities. The study investigated neural reward and loss processing and whether age, pubertal status and the social constructs of gender and race predicted individual differences in reward- and loss-related brain function. One hundred and twenty-eight primarily low-income adolescents (mean age: 15.9 years, 75% African American) from urban environments completed a modified monetary incentive delay task during functional magnetic resonance imaging (fMRI). Consistent with the previous research, reward and loss anticipation recruited similar motivational circuitry including striatal, insular, thalamic and supplementary motor areas. Race and gender were not associated with reward- or loss-related neural reactivity. Age and pubertal development were associated with differences in neural reactivity to reward and loss, suggesting that older and more mature adolescents had increased activity in sensory and motivational circuits, but decreased activity in regions responsible for error detection and behavior modification.     

Weak reward source memory in depression reflects blunted activation of VTA/SN and parahippocampus

Reward responses in the medial temporal lobes and dopaminergic midbrain boost episodic memory formation in healthy adults, and weak memory for emotionally positive material in depression suggests this mechanism may be dysfunctional in major depressive disorder (MDD). To test this hypothesis, we performed a study in which unmedicated adults with MDD and healthy controls encoded drawings paired with reward or zero tokens during functional magnetic resonance imaging. In a recognition test, participants judged whether drawings were previously associated with the reward token (‘reward source’) or the zero token (‘zero source’). Unlike controls, depressed participants failed to show better memory for drawings from the reward source vs the zero source. Consistent with predictions, controls also showed a stronger encoding response to reward tokens vs zero tokens in the right parahippocampus and dopaminergic midbrain, whereas the MDD group showed the opposite pattern—stronger responses to zero vs reward tokens—in these regions. Differential activation of the dopaminergic midbrain by reward vs zero tokens was positively correlated with the reward source memory advantage in controls, but not depressed participants. These data suggest that weaker memory for positive material in depression reflects blunted encoding responses in the dopaminergic midbrain and medial temporal lobes.     

Surprise beyond prediction error

Surprise drives learning. Various neural “prediction error” signals are believed to underpin surprise‐based reinforcement learning. Here, we report a surprise signal that reflects reinforcement learning but is neither un/signed reward prediction error (RPE) nor un/signed state prediction error (SPE). To exclude these alternatives, we measured surprise responses in the absence of RPE and accounted for a host of potential SPE confounds. This new surprise signal was evident in ventral striatum, primary sensory cortex, frontal poles, and amygdala. We interpret these findings via a normative model of surprise. Hum Brain Mapp 35:4805–4814, 2014. © 2014 The Authors. Human Brain Mapping Published by Wiley Periodicals, Inc.     

Neural changes in reward processing following approach-avoidance training for depression

Altered approach motivation is hypothesized to be critical for the maintenance of depression. Computer-administered approach-avoidance training programs to increase approach action tendencies toward positive stimuli produce beneficial outcomes. However, there have been few studies examining neural changes following approach-avoidance training. Participants with major depressive disorder were randomized to an approach-avoidance training (AAT) manipulation intended to increase approach tendencies for positive social cues (n = 13) or a control procedure (n = 15). We examined changes in neural activation (primary outcome) and connectivity patterns using Group Iterative Multiple Model Estimation during a social reward anticipation task (exploratory). A laboratory-based social affiliation task was also administered following the manipulation to measure affect during anticipation of real-world social activity. Individuals in the AAT group demonstrated increased activation in reward processing regions during social reward anticipation relative to the control group from pre- to post-training. Following training, connectivity patterns across reward regions were observed in the full sample and connectivity between the medial prefrontal cortex and caudate was associated with anticipatory positive affect before the social interaction. Preliminary evidence of differential connectivity patterns between the two groups also emerged. Results support models whereby modifying approach-oriented behavioral tendencies with computerized training lead to alterations in reward circuitry ({"type":"clinical-trial","attrs":{"text":"NCT02330744","term_id":"NCT02330744"}}NCT02330744).     

Cognitive strategies regulate fictive,but not reward prediction error signals in a sequential investment task

Computational models of reward processing suggest that foregone or fictive outcomes serve as important information sources for learning and augment those generated by experienced rewards (e.g. reward prediction errors). An outstanding question is how these learning signals interact with top‐down cognitive influences, such as cognitive reappraisal strategies. Using a sequential investment task and functional magnetic resonance imaging, we show that the reappraisal strategy selectively attenuates the influence of fictive, but not reward prediction error signals on investment behavior; such behavioral effect is accompanied by changes in neural activity and connectivity in the anterior insular cortex, a brain region thought to integrate subjective feelings with high‐order cognition. Furthermore, individuals differ in the extent to which their behaviors are driven by fictive errors versus reward prediction errors, and the reappraisal strategy interacts with such individual differences; a finding also accompanied by distinct underlying neural mechanisms. These findings suggest that the variable interaction of cognitive strategies with two important classes of computational learning signals (fictive, reward prediction error) represent one contributing substrate for the variable capacity of individuals to control their behavior based on foregone rewards. These findings also expose important possibilities for understanding the lack of control in addiction based on possibly foregone rewarding outcomes. Hum Brain Mapp 35:3738–3749, 2014. © 2013 The Authors. Human Brain Mapping Published by Wiley Periodicals, Inc.     

Neural correlates of reward and loss sensitivity in psychopathy

Psychopathy is a personality disorder associated with callous and impulsive behavior and criminal recidivism. It has long been theorized that psychopaths have deficits in processing reward and punishment. Here, we use structural and functional magnetic resonance imaging to examine the neural correlates of reward and loss sensitivity in a group of criminal psychopaths. Forty-one adult male prison inmates (n = 18 psychopaths and n = 23 non-psychopaths) completed a functional magnetic resonance imaging task involving the gain or loss of money. Across the entire sample of participants, monetary gains elicited robust activation within the ventral striatum (VS). Although psychopaths and non-psychopaths did not significantly differ with respect to overall levels of VS response to reward vs loss, we observed significantly different correlations between VS responses and psychopathy severity within each group. Volumetric analyses of striatal subregions revealed a similar pattern of correlations, specifically for the right accumbens area within VS. In a separate sample of inmates (n = 93 psychopaths and n = 117 non-psychopaths) who completed a self-report measure of appetitive motivation, we again found that the correlation with psychopathy severity differed between groups. These convergent results offer novel insight into the neural substrates of reward and loss processing in psychopathy.     

Adaptive neural reward processing during anticipation and receipt of monetary rewards in mindfulness meditators

Reward seeking is ubiquitous and adaptive in humans. But excessive reward seeking behavior, such as chasing monetary rewards, may lead to diminished subjective well-being. This study examined whether individuals trained in mindfulness meditation show neural evidence of lower susceptibility to monetary rewards. Seventy-eight participants (34 meditators, 44 matched controls) completed the monetary incentive delay task while undergoing functional magnetic resonance imaging. The groups performed equally on the task, but meditators showed lower neural activations in the caudate nucleus during reward anticipation, and elevated bilateral posterior insula activation during reward anticipation. Meditators also evidenced reduced activations in the ventromedial prefrontal cortex during reward receipt compared with controls. Connectivity parameters between the right caudate and bilateral anterior insula were attenuated in meditators during incentive anticipation. In summary, brain regions involved in reward processing—both during reward anticipation and receipt of reward—responded differently in mindfulness meditators than in nonmeditators, indicating that the former are less susceptible to monetary incentives.     

Separate neural representations of prediction error valence and surprise: Evidence from an fMRI meta‐analysis

Learning occurs when an outcome differs from expectations, generating a reward prediction error signal (RPE). The RPE signal has been hypothesized to simultaneously embody the valence of an outcome (better or worse than expected) and its surprise (how far from expectations). Nonetheless, growing evidence suggests that separate representations of the two RPE components exist in the human brain. Meta‐analyses provide an opportunity to test this hypothesis and directly probe the extent to which the valence and surprise of the error signal are encoded in separate or overlapping networks. We carried out several meta‐analyses on a large set of fMRI studies investigating the neural basis of RPE, locked at decision outcome. We identified two valence learning systems by pooling studies searching for differential neural activity in response to categorical positive‐versus‐negative outcomes. The first valence network (negative > positive) involved areas regulating alertness and switching behaviours such as the midcingulate cortex, the thalamus and the dorsolateral prefrontal cortex whereas the second valence network (positive > negative) encompassed regions of the human reward circuitry such as the ventral striatum and the ventromedial prefrontal cortex. We also found evidence of a largely distinct surprise‐encoding network including the anterior cingulate cortex, anterior insula and dorsal striatum. Together with recent animal and electrophysiological evidence this meta‐analysis points to a sequential and distributed encoding of different components of the RPE signal, with potentially distinct functional roles.     

Altered neural function during episodic memory encoding and retrieval in major depression

Memory impairments are common in major depression. Neural processing during non‐emotional episodic memory in depressed patients has only sparsely been investigated, since the majority of studies have focused on emotional stimuli. The aim of this study was to explore neural correlates of episodic memory in depressive patients and to assess brain regions related to subsequent memory performance. Forty‐six participants (23 depressed patients) performed a non‐emotional episodic memory encoding and retrieval task while brain activation was measured with functional magnetic resonance imaging. Patients with depression showed decreased activation in the right prefrontal cortex and right cingulate cortex during memory encoding, but increased activation in the right inferior frontal gyrus (IFG) during recognition memory. While a strong association between hippocampal and parahippocampal activation during memory encoding with subsequent memory performance became evident in healthy controls, this relationship was absent in patients with depression. Taken together, these findings demonstrate that memory related brain regions are affected in their appropriate functioning during memory encoding in depressed patients. Therefore, patients with depression may rely to a greater degree on other brain regions such as the IFG during episodic memory retrieval. Hum Brain Mapp 35:4293–4302, 2014. © 2014 Wiley Periodicals, Inc .     

A reward prediction error for charitable donations reveals outcome orientation of donators

The motives underlying prosocial behavior, like charitable donations, can be related either to actions or to outcomes. To address the neural basis of outcome orientation in charitable giving, we asked 33 subjects to make choices affecting their own payoffs and payoffs to a charity organization, while being scanned by functional magnetic resonance imaging (fMRI). We experimentally induced a reward prediction error (RPE) by subsequently discarding some of the chosen outcomes. Co-localized to a nucleus accumbens BOLD signal corresponding to the RPE for the subject''s own payoff, we observed an equivalent RPE signal for the charity''s payoff in those subjects who were willing to donate. This unique demonstration of a neuronal RPE signal for outcomes exclusively affecting unrelated others indicates common brain processes during outcome evaluation for selfish, individual and nonselfish, social rewards and strongly suggests the effectiveness of outcome-oriented motives in charitable giving.     

Pain modulates neural responses to reward in the medial prefrontal cortex

Pain has been found to promote reward‐seeking behaviors, which might be a consequence of modulated brain activities in the reward neural circuitry in a painful state. The present study investigated how pain affected reward processing and reward‐related neural activities using fMRI technique. A total of 50 healthy participants were recruited and used for data analyses, with half being treated with topical capsaicin cream and the other half with hand cream (treatment: pain or control). The participants were asked to perform a card‐guessing game when their brain activities responding to feedbacks (outcome: win or loss) were recorded. Behavioral results showed that participants in pain group overestimated their correct choices in the card‐guess game. Whole‐brain fMRI analysis revealed that the main effect of outcome (win vs. loss) activated a typical network of the reward neural circuitry, including the medial prefrontal cortex (mPFC) and the bilateral nucleus accumbens (NAcc). Importantly, the region of interest analysis revealed a significant interaction of treatment and outcome in the mPFC, with increased mPFC neural activity responding to win outcome in pain condition. Moreover, the functional connectivity between the mPFC and the NAcc was decreased in pain condition. We conclude that the pain‐induced modulation of the mPFC activity could result in alterations of both the emotional response to and the cognitive evaluation of reward.     

Impact of aging on frontostriatal reward processing

Healthy aging is associated with a progressive decline across a range of cognitive functions. An important factor underlying this decline may be the age‐related impairment in stimulus–reward processing. Several studies have investigated age‐related effects, but compared young versus old subjects. This is the first study to investigate the effect of aging on brain activation during reward processing within a continuous segment of the adult life span. We scanned 49 healthy adults aged 40–70 years, using functional MRI. We adopted a simple reward task, which allowed separate evaluation of neural responses to reward anticipation and receipt. The effect of reward on performance accuracy and speed was not related to age, indicating that all subjects could perform the task correctly. We identified a whole‐brain significant age‐related decline of ventral striatum activation during reward anticipation as compared to neutral anticipation. Importantly, the specificity of this finding was underscored by the observation that there was no general decline in activation during anticipation. Activation in the ventral striatum increased with age during reward receipt as compared to receiving neutral outcome. Finally, activation in the ventromedial prefrontal cortex during outcome was not affected by age. Our data demonstrate that the typical shift in striatal activation from reward receipt to reward anticipation in young adults disappears with healthy aging. These changes are consistent the well‐ocumented age‐related decline of striatal dopamine availability, and may provide a stepping stone for further research of age‐related neurodegenerative diseases. Hum Brain Mapp 36:2305–2317, 2015. © 2015 Wiley Periodicals, Inc.     

Separate brain regions code for salience vs. valence during reward prediction in humans

Predicting rewards and avoiding aversive conditions is essential for survival. Recent studies using computational models of reward prediction implicate the ventral striatum in appetitive rewards. Whether the same system mediates an organism's response to aversive conditions is unclear. We examined the question using fMRI blood oxygen level-dependent measurements while healthy volunteers were conditioned using appetitive and aversive stimuli. The temporal difference learning algorithm was used to estimate reward prediction error. Activations in the ventral striatum were robustly correlated with prediction error, regardless of the valence of the stimuli, suggesting that the ventral striatum processes salience prediction error. In contrast, the orbitofrontal cortex and anterior insula coded for the differential valence of appetitive/aversive stimuli. Given its location at the interface of limbic and motor regions, the ventral striatum may be critical in learning about motivationally salient stimuli, regardless of valence, and using that information to bias selection of actions.     

Individual associations of adolescent alcohol use disorder versus cannabis use disorder symptoms in neural prediction error signaling and the response to novelty

Two of the most commonly used illegal substances by adolescents are alcohol and cannabis. Alcohol use disorder (AUD) and cannabis use disorder (CUD) are associated with poorer decision-making in adolescents. In adolescents, level of AUD symptomatology has been negatively associated with striatal reward responsivity. However, little work has explored the relationship with striatal reward prediction error (RPE) representation and the extent to which any augmentation of RPE by novel stimuli is impacted. One-hundred fifty-one adolescents participated in the Novelty Task while undergoing functional magnetic resonance imaging (fMRI). In this task, participants learn to choose novel or non-novel stimuli to gain monetary reward. Level of AUD symptomatology was negatively associated with both optimal decision-making and BOLD response modulation by RPE within striatum and regions of prefrontal cortex. The neural alterations in RPE representation were particularly pronounced when participants were exploring novel stimuli. Level of CUD symptomatology moderated the relationship between novelty propensity and RPE representation within inferior parietal lobule and dorsomedial prefrontal cortex. These data expand on an emerging literature investigating individual associations of AUD symptomatology levels versus CUD symptomatology levels and RPE representation during reinforcement processing and provide insight on the role of neuro-computational processes underlying reinforcement learning/decision-making in adolescents.