Temporal and probabilistic discounting of rewards in children and adolescents: effects of age and ADHD symptoms

This study investigated whether age and ADHD symptoms affected choice preferences in children and adolescents when they chose between (1) small immediate rewards and larger delayed rewards and (2) small certain rewards and larger probabilistic uncertain rewards. A temporal discounting (TD) task and a probabilistic discounting (PD) task were used to measure the degree to which the subjective value of a large reward decreased as one had to wait longer for it (TD), and as the probability of obtaining it decreased (PD). Rewards used were small amounts of money. In the TD task, the large reward (10 cents) was delayed by between 0 and 30s, and the immediate reward varied in magnitude (0-10 cents). In the PD task, receipt of the large reward (10 cents) varied in likelihood, with probabilities of 0, 0.25, 0.5, 0.75, and 1.0 used, and the certain reward varied in magnitude (0-10 cents). Age and diagnostic group did not affect the degree of PD of rewards: All participants made choices so that total gains were maximized. As predicted, young children, aged 6-11 years (n = 25) demonstrated steeper TD of rewards than adolescents, aged 12-17 years (n = 21). This effect remained significant even when choosing the immediate reward did not shorten overall task duration. This, together with the lack of interaction between TD task version and age, suggests that steeper discounting in young children is driven by reward immediacy and not by delay aversion. Contrary to our predictions, participants with ADHD (n = 22) did not demonstrate steeper TD of rewards than controls (n = 24). These results raise the possibility that strong preferences for small immediate rewards in ADHD, as found in previous research, depend on factors such as total maximum gain and the use of fixed versus varied delay durations. The decrease in TD as observed in adolescents compared to children may be related to developmental changes in the (dorsolateral) prefrontal cortex. Future research needs to investigate these possibilities.     

Caudate responses to reward anticipation associated with delay discounting behavior in healthy youth

BackgroundChoices requiring delay of gratification made during adolescence can have significant impact on life trajectory. Willingness to delay gratification can be measured using delay discounting tasks that require a choice between a smaller immediate reward and a larger delayed reward. Individual differences in the subjective value of delayed rewards are associated with risk for development of psychopathology including substance abuse. The neurobiological underpinnings related to these individual differences early in life are not fully understood. Using functional magnetic resonance imaging (fMRI), we tested the hypothesis that individual differences in delay discounting behavior in healthy youth are related to differences in responsiveness to potential reward.MethodNineteen 10–14 year-olds performed a monetary incentive delay task to assess neural sensitivity to potential reward and a questionnaire to measure discounting of future monetary rewards.ResultsLeft ventromedial caudate activation during anticipation of potential reward was negatively correlated with delay discounting behavior. There were no regions where brain responses during notification of reward outcome were associated with discounting behavior.ConclusionsBrain activation during anticipation of potential reward may serve as a marker for individual differences in ability or willingness to delay gratification in healthy youth.     

Hypothalamic-Extended Amygdala Circuit Regulates Temporal Discounting

Choice behavior is characterized by temporal discounting, i.e., preference for immediate rewards given a choice between immediate and delayed rewards. Agouti-related peptide (AgRP)-expressing neurons located in the arcuate nucleus of the hypothalamus (ARC) regulate food intake and energy homeostasis, yet whether AgRP neurons influence choice behavior and temporal discounting is unknown. Here, we demonstrate that motivational state potently modulates temporal discounting. Hungry mice (both male and female) strongly preferred immediate food rewards, yet sated mice were largely indifferent to reward delay. More importantly, selective optogenetic activation of AgRP-expressing neurons or their axon terminals within the posterior bed nucleus of stria terminalis (BNST) produced temporal discounting in sated mice. Furthermore, activation of neuropeptide Y (NPY) type 1 receptors (Y1Rs) within the BNST is sufficient to produce temporal discounting. These results demonstrate a profound influence of hypothalamic signaling on temporal discounting for food rewards and reveal a novel circuit that determine choice behavior.SIGNIFICANCE STATEMENT Temporal discounting is a universal phenomenon found in many species, yet the underlying neurocircuit mechanisms are still poorly understood. Our results revealed a novel neural pathway from agouti-related peptide (AgRP) neurons in the hypothalamus to the bed nucleus of stria terminalis (BNST) that regulates temporal discounting in decision-making.     

Hippocampus,delay discounting,and vicarious trial‐and‐error

In decision‐making, an immediate reward is usually preferred to a delayed reward, even if the latter is larger. We tested whether the hippocampus is necessary for this form of temporal discounting, and for vicarious trial‐and‐error at the decision point. Rats were trained on a recently developed, adjustable delay‐discounting task (Papale et al. (2012) Cogn Affect Behav Neurosci 12:513–526), which featured a choice between a small, nearly immediate reward, and a larger, delayed reward. Rats then received either hippocampus or sham lesions. Animals with hippocampus lesions adjusted the delay for the larger reward to a level similar to that of sham‐lesioned animals, suggesting a similar valuation capacity. However, the hippocampus lesion group spent significantly longer investigating the small and large rewards in the first part of the sessions, and were less sensitive to changes in the amount of reward in the large reward maze arm. Both sham‐ and hippocampus‐lesioned rats showed a greater amount of vicarious trial‐and‐error on trials in which the delay was adjusted. In a nonadjusting version of the delay discounting task, animals with hippocampus lesions showed more variability in their preference for a larger reward that was delayed by 10 s compared with sham‐lesioned animals. To verify the lesion behaviorally, rat were subsequently trained on a water maze task, and rats with hippocampus lesions were significantly impaired compared with sham‐lesioned animals. The findings on the delay discounting tasks suggest that damage to the hippocampus may impair the detection of reward magnitude. © 2014 Wiley Periodicals, Inc.     

Cocaine-induced impulsive choices are accompanied by impaired delay-dependent anticipatory activity in basolateral amygdala

Addicts and drug-experienced animals have decision-making deficits in delayed reinforcement choice task, in which they prefer small immediate rewards over large delayed rewards. Here, we show evidence that this deficit is accompanied by changed coding of delay length in the basolateral amygdala (BLA). A subset of neurons in BLA demonstrated delay-dependent anticipatory activity (either increase or decrease as a function of delay to reward) in naive rats. After 30 days of withdrawal from chronic cocaine treatment (30 mg/kg/day for 10 days ip), the proportion of delay-dependent anticipatory neurons reduced, whereas delay-dependent activity in response to elapsed delay after reward delivery increased, both in the proportion of delay-dependent neurons and in the extent of delay dependence. Cocaine exposure increased, instead of decreased, BLA neuronal expectation for different reward magnitudes. These results indicate that BLA is critical for representing and maintaining the information of delayed reward before its delivery, and cocaine exposure may affect decision-making by impairing perception of delay instead of the ability to assess the differences in reward size.     

Planning to make economic decisions in the future,but choosing impulsively now: are preference reversals related to symptoms of ADHD and depression?

A preference for smaller immediate rewards over larger delayed rewards (delay discounting, DD) is common in attention deficit hyperactivity disorder (ADHD), but rarely investigated in depression. Whether this preference is due to sensitivity to reward immediacy or delay aversion remains unclear. To investigate this, we examined whether ADHD and depressive symptoms are associated with preference reversals: a switch from smaller immediate rewards to larger delayed rewards when smaller rewards are also delayed. We also examined whether these symptoms differentially affect DD of losses. In Study 1 undergraduates completed a questionnaire about ADHD symptoms, and performed a hypothetical DD task. In the NOW condition, participants were presented with choices between a small reward available today and a large reward available after one year. In the FUTURE condition both rewards were delayed with +1 year. In Study 2 undergraduates completed questionnaires about ADHD and depressive symptoms and performed a DD task with gains and losses. Participants showed preference reversals in both studies and tasks. Losses were less steeply discounted than gains. ADHD and depressive symptoms did not influence these effects. Depressive symptoms, but not ADHD symptoms, were associated with less economic choices in general. These findings suggest that impulsive choice in depression is not explained by sensitivity to reward immediacy. Copyright © 2016 John Wiley & Sons, Ltd.     

Dissociable contributions of the ventral hippocampus and orbitofrontal cortex to decision‐making with a delayed or uncertain outcome

In this study, we examined how risk and delay influence rats' decision‐making, and the role of the ventral hippocampus (VHC) and orbitofrontal cortex (OFC) in the valuation of these two factors. We used a touchscreen testing method in which rats with VHC lesions, OFC lesions and sham control surgery made choices in two decision‐making tasks. In the delay discounting task, rats chose between two visual stimuli, one of which indicated a small, immediate reward, and the other of which indicated a large, delayed reward. In the probability discounting task, two stimuli indicated, instead, a small, certain reward or a large, uncertain reward. The two lesion groups showed a double dissociation with respect to the two tasks. Rats with VHC lesions were intolerant of delay, and were strongly biased towards the small, immediate reward. However, the same rats were indistinguishable from sham controls in the probability discounting task. The opposite pattern was observed for rats with OFC lesions; they performed normally in the delay discounting task, but showed a reduced tolerance for uncertainty as compared with sham‐operated controls. These data support the conclusion that the VHC and OFC contribute differentially to decision‐making that involves delayed or uncertain outcomes. This provides a means for understanding the neural basis of a range of neurological and psychiatric patients who show impaired decision‐making and executive dysfunction.     

Delay discounting and future-directed thinking in anhedonic individuals

Anhedonia (lack of reactivity to pleasurable stimuli) and a negatively skewed view of the future are important components of depression that could affect economic decisions in depressed individuals. Delay discounting paradigms might be useful for probing putative affective and cognitive underpinnings of such decisions. As a first step to evaluate whether difficulties experiencing pleasure might affect delay discounting, 36 undergraduate students with varying levels of anhedonia performed a delay discounting task in which they made choices between a small immediate and larger future monetary reward. Increasing levels of anhedonia (Snaith-Hamilton Pleasure Scale) were negatively associated with delay discounting rate, indicating that anhedonic individuals tended to choose the larger, albeit delayed reward. These correlations remained after controlling for variables previously linked to delay discounting (working memory capacity and impulsivity) and pessimistic future-directed thinking. The current findings provide preliminary evidence indicating that anhedonic individuals make less myopic decisions about their future, possibly due to their decreased responsiveness to immediate rewards.     

Effects of serotonin on delay discounting for rewards--an application for understanding of pathophysiology in psychiatric disorders

In our daily life, we constantly make such choices between actions leading to rewards of various sizes after different delays. "Delay discounting" is a theoretical concept in which the "value" of reward R after delay. A steep rate of discounting results in impulsive choice, defined by an abnormally frequent choice of the more immediate reward. Our behavioral and neuroimaging results suggest that serotonin may adjust the rate of delayed reward discounting via the modulation of striatum in cortico-basal ganglia circuits in human. Our proposed role of serotonin may explain certain aspects of impulsivity in psychiatric disorders such as major depression, panic disorder or obsessive-compulsive disorder, that are known to effectively relieve symptoms by selective serotonin reuptake inhibitors. Future experiments using delayed reward paradigms could be designed to study impulsivity in these patients.     

Dopamine, corticostriatal connectivity, and intertemporal choice

Value-based decisions optimize behavioral outcomes. Because delayed rewards are discounted, an increased tendency to choose smaller, immediate rewards can lead to suboptimal choice. Steep discounting of delayed rewards (impulsivity) characterizes subjects with frontal lobe damage and behavioral disorders including substance abuse. Correspondingly, animal studies and indirect evidence in humans suggest that lower dopamine in the frontal cortex contributes to steeper discounting by impairing corticostriatal function. To test this hypothesis directly, we performed a randomized, double-blind, counterbalanced, placebo-controlled study in which we administered the brain penetrant catechol-O-methyltransferase inhibitor tolcapone or placebo to healthy subjects performing a delay discounting task. Tolcapone significantly increased choice of delayed monetary rewards, and this tolcapone-induced increase covaried with increased BOLD activity in the left ventral putamen and anterior insula. Tolcapone also changed corticostriatal connectivity: specifically, by inducing a decrease in the coherence between ventral putamen and pregenual cingulate cortex. These results indicate that raising cortical dopamine levels attenuates impulsive choice by changing corticostriatal function.     

Nucleus accumbens lesions decrease sensitivity to rapid changes in the delay to reinforcement

Both humans and non-humans discount the value of rewards that are delayed or uncertain, and individuals that discount delayed rewards at a relatively high rate are considered impulsive. To investigate the neural mechanisms that mediate delay discounting, the present study examined the effects of excitotoxic lesions of the nucleus accumbens (NAC) on discounting of reward value by delay and probability. Rats were trained on delay (n=24) or probability discounting (n=24) tasks. Following training, excitotoxic lesions of the NAC were made by intracranial injections of 0.5 microl 0.15 M quinolinic acid (n=12) or vehicle (n=12) aimed at the NAC (AP +1.6, ML +/-1.5, DV -7.1). NAC lesions did not alter performance in animals tested with a constant delay (4s) or probability (0.4) of reinforcement. However, when tested with between session changes in the delay (0, 1, 2, 4, and 8s) of reinforcement, the lesioned rats had flatter discount curves than the sham group, indicating that they were less sensitive to frequent changes in the delay to reward. In contrast, the NAC lesions did not affect discounting of probabilistic rewards. NAC lesions impaired the ability to adapt to frequent between session changes in the delay to reward but did not increase or decrease discounting when the delay was held constant across sessions. NAC lesions may disrupt the ability of the animals to predict the timing of delayed rewards when the delay to reward is changed frequently.     

Reward sensitivity in impulsivity

Impulsive individuals choose immediate small over delayed larger rewards, suggesting reward hypersensitivity. Single-unit studies have shown increased ventral tegmental activity to rewards and reward predictors and decreased activity when predicted rewards are withheld. The orbitofrontal ventral tegmental cortical target also responds to reward and expectation in single-unit and neuroimaging studies. The anterior P2a event-related potential component is a proposed index of reward-related orbitofrontal activity. In this reward prediction study in high and low impulsive subjects, the P2a localized to orbitofrontal cortex and was largest to non-predicted rewards and smallest in the absence of predicted rewards in subjects higher on self-reported impulsiveness, consistent with a P2a index of orbitofrontal reward processing and with reward hypersensitivity in impulsivity.     

The neurobiology of intertemporal choice: insight from imaging and lesion studies

People are frequently faced with intertemporal choices, i.e., choices differing in the timing of their consequences, preferring smaller rewards available immediately over larger rewards delivered after a delay. The inability to forgo sooner gratification to favor delayed reward (e.g., impulsivity) has been related to several pathological conditions characterized by poor self-control, including drug addiction and obesity. Comparative and functional human studies have implicated a network of brain areas involved in intertemporal choice, including the medial portion of the orbitofrontal cortex (mOFC). Moreover, damage to this cortical area increases preference for immediate gratification in intertemporal decisions. Here, we review recent neuroscientific studies concerning intertemporal choice, suggesting that the mOFC contributes to preference for delayed rewards, either by computing the value of future outcomes (i.e., valuation), or by enabling people to imagine and represent future rewards and their consequences (e.g., prospection).     

The neuroeconomics of nicotine dependence: a preliminary functional magnetic resonance imaging study of delay discounting of monetary and cigarette rewards in smokers

Neuroeconomics integrates behavioral economics and cognitive neuroscience to understand the neurobiological basis for normative and maladaptive decision making. Delay discounting is a behavioral economic index of impulsivity that reflects capacity to delay gratification and has been consistently associated with nicotine dependence. This preliminary study used functional magnetic resonance imaging to examine delay discounting for money and cigarette rewards in 13 nicotine dependent adults. Significant differences between preferences for smaller immediate rewards and larger delayed rewards were evident in a number of regions of interest (ROIs), including the medial prefrontal cortex, anterior insular cortex, middle temporal gyrus, middle frontal gyrus, and cingulate gyrus. Significant differences between money and cigarette rewards were generally lateralized, with cigarette choices associated with left hemisphere activation and money choices associated with right hemisphere activation. Specific ROI differences included the posterior parietal cortex, medial and middle frontal gyrus, ventral striatum, temporoparietal cortex, and angular gyrus. Impulsivity as measured by behavioral choices was significantly associated with both individual ROIs and a combined ROI model. These findings provide initial evidence in support of applying a neuroeconomic approach to understanding nicotine dependence.     

Neural systems implicated in delayed and probabilistic reinforcement.

This review considers the theoretical problems facing agents that must learn and choose on the basis of reward or reinforcement that is uncertain or delayed, in implicit or procedural (stimulus-response) representational systems and in explicit or declarative (action-outcome-value) representational systems. Individual differences in sensitivity to delays and uncertainty may contribute to impulsivity and risk taking. Learning and choice with delayed and uncertain reinforcement are related but in some cases dissociable processes. The contributions to delay and uncertainty discounting of neuromodulators including serotonin, dopamine, and noradrenaline, and of specific neural structures including the nucleus accumbens core, nucleus accumbens shell, orbitofrontal cortex, basolateral amygdala, anterior cingulate cortex, medial prefrontal (prelimbic/infralimbic) cortex, insula, subthalamic nucleus, and hippocampus are examined.     

Greater impulsivity is associated with decreased brain activation in obese women during a delay discounting task

Impulsivity and poor inhibitory control are associated with higher rates of delay discounting (DD), or a greater preference for smaller, more immediate rewards at the expense of larger, but delayed rewards. Of the many functional magnetic resonance imaging (fMRI) studies of DD, few have investigated the correlation between individual differences in DD rate and brain activation related to DD trial difficulty, with difficult DD trials expected to activate putative executive function brain areas involved in impulse control. In the current study, we correlated patterns of brain activation as measured by fMRI during difficult vs. easy trials of a DD task with DD rate (k) in obese women. Difficulty was defined by how much a reward choice deviated from an individual’s ‘indifference point’, or the point where the subjective preference for an immediate and a delayed reward was approximately equivalent. We found that greater delay discounting was correlated with less modulation of activation in putative executive function brain areas, such as the middle and superior frontal gyri and inferior parietal lobule, in response to difficult compared to easy DD trials. These results support the suggestion that increased impulsivity is associated with deficient functioning of executive function areas of the brain.     

Delay discounting in mild cognitive impairment

Introduction: Patients with mild cognitive impairment (MCI) may make suboptimal decisions particularly in complex situations, and this could be due to temporal discounting, the tendency to prefer immediate rewards over delayed but larger rewards. The present study proposes to evaluate intertemporal preferences in MCI patients as compared to healthy controls. Method: Fifty-five patients with MCI and 57 healthy controls underwent neuropsychological evaluation and a delay discounting questionnaire, which evaluates three parameters: hyperbolic discounting (k), the percentage of choices for delayed and later rewards (%LL), and response consistency (Acc). Results: No significant differences were found in the delay discounting questionnaire between MCI patients and controls for the three reward sizes considered, small, medium, and large, using both k and %LL parameters. There were also no differences in the response consistency, Acc, between the two groups. Conclusions: Patients with MCI perform similarly to healthy controls in a delay discounting task. Memory deficits do not notably affect intertemporal preferences.     

Lost in transition: aging-related changes in executive control by the medial prefrontal cortex

Neural correlates of aging in the medial prefrontal cortex (mPFC) were studied using an operant delayed response task. The task used blocks of trials with memory-guided (delayed alternation) and visually-guided (stimulus-response) responding. Older rats (24 months) performed at a slow pace compared with younger rats (6 months). They wasted time engaged in nonessential behaviors (e.g., licking on spouts beyond the period of reward delivery) and were slow to respond at the end of the delay period. Aged mPFC neurons showed normal spatial processing. They differed from neurons in younger rats by having reduced modulations by imperative stimuli indicating reward availability and reduced activity associated with response latencies for reward collection. Older rats showed reduced sensitivity to imperative stimuli at three levels of neural activity: reduced fractions of neurons with changes in firing rate around the stimulus, reduced correlation over neurons at the time of the stimulus as measured with analysis of population activity, and reduced amplitudes of event-related fluctuations in intracortical field potentials at the time of the imperative stimulus. Our findings suggest that aging alters the encoding of time-sensitive information and impairs the ability of prefrontal networks to keep subjects "on task."     

A Neural Mechanism for Reward Discounting: Insights from Modeling Hippocampal–Striatal Interactions

Decision-making often requires taking into consideration immediate gains as well as delayed rewards. Studies of behavior have established that anticipated rewards are discounted according to a decreasing hyperbolic function. Although mathematical explanations for reward delay discounting have been offered, little has been proposed in terms of neural network mechanisms underlying discounting. There has been much recent interest in the potential role of the hippocampus. Here, we demonstrate that a previously established neural network model of hippocampal region CA3 contains a mechanism that could explain discounting in downstream reward-prediction systems (e.g., basal ganglia). As part of its normal function, the model forms codes for stimuli that are similar to future predicted stimuli. This similarity provides a means for reward predictions associated with future stimuli to influence current decision-making. Simulations show that this “predictive similarity” decreases as the stimuli are separated in time, at a rate that is consistent with hyperbolic discounting.     

Studying the relation between temporal reward discounting tasks used in populations with ADHD: A factor analysis

Background: This study aimed at investigating the relationship between tasks that have been used in attention deficit hyperactivity disorder (ADHD) to measure choices between smaller immediate and larger delayed rewards: real and hypothetical temporal discounting tasks, and single‐choice paradigms. Methods: Participants were 55 undergraduate psychology students. Tasks included a real and hypothetical version of a temporal discounting (TD) task with choices between a large reward (10 cents) after delays up to 60 seconds, and smaller immediate rewards (2–8 cents); two versions of a hypothetical temporal discounting task with choices between a large reward ($100) after delays up to 120 months, and smaller immediate rewards ($1–$95); a Choice Delay Task with choices between one point now and two points after 30 seconds (one point is worth five cents). Results: Correlation analyses showed that the real and the hypothetical TD tasks with 10 cents were very strongly associated. However, the hypothetical TD tasks with $100 did not correlate with either the real or the hypothetical TD task with 10 cents. Principal component analysis extracted two components: one for small amounts and short delays, and a second one for large rewards and long delays. Conclusions: Temporal reward discounting is not a uniform construct. Functional brain imaging research could shed more light on unique brain activation patterns associated with different forms of temporal reward discounting. Copyright © 2010 John Wiley & Sons, Ltd.