Responses to reward in monkey dorsal and ventral striatum

Summary The sources of input and the behavioral effects of lesions and drug administration suggest that the striatum participates in motivational processes. We investigated the activity of single striatal neurons of monkeys in response to reward delivered for performing in a go-nogo task. A drop of liquid was given each time the animal correctly executed or withheld an arm movement in reaction to a visual stimulus. Of 1593 neurons, 115 showed increased activity in response to delivery of liquid reward in both go and nogo trials. Responding neurons were predominantly located in dorsal and ventromedial parts of anterior putamen, in dorsal and ventral caudate, and in nucleus accumbens. They were twice as frequent in ventral as compared to dorsal striatal areas. Responses occurred at a median latency of 337 ms and lasted for 525 ms, with insignificant differences between dorsal and ventral striatum. Reward responses differed from activity recorded in the face area of posterior putamen which varied synchronously with individual mouth movements. Responses were directly related to delivery of primary liquid reward and not to auditory stimuli associated with it. Most of them also occurred when reward was delivered outside of the task. These results demonstrate that neurons of dorsal and particularly ventral striatum are involved in processing information concerning the attribution of primary reward.     

Neuronal activity in the rodent dorsal striatum in sequential navigation: separation of spatial and reward responses on the multiple T task

The striatum plays an important role in "habitual" learning and memory and has been hypothesized to implement a reinforcement-learning algorithm to select actions to perform given the current sensory input. Many experimental approaches to striatal activity have made use of temporally structured tasks, which imply that the striatal representation is temporal. To test this assumption, we recorded neurons in the dorsal striatum of rats running a sequential navigation task: the multiple T maze. Rats navigated a sequence of four T maze turns to receive food rewards delivered in two locations. The responses of neurons that fired phasically were examined. Task-responsive phasic neurons were active as rats ran on the maze (maze-responsive) or during reward receipt (reward-responsive). Neither mazenor reward-responsive neurons encoded simple motor commands: maze-responses were not well correlated with the shape of the rat's path and most reward-responsive neurons did not fire at similar rates at both food-delivery sites. Maze-responsive neurons were active at one or more locations on the maze, but these responses did not cluster at spatial landmarks such as turns. Across sessions the activity of maze-responsive neurons was highly correlated when rats ran the same maze. Maze-responses encoded the location of the rat on the maze and imply a spatial representation in the striatum in a task with prominent spatial demands. Maze-responsive and reward-responsive neurons were two separate populations, suggesting a divergence in striatal information processing of navigation and reward.     

Encoding of action history in the rat ventral striatum

In a dynamic environment, animals need to update information about the rewards expected from their alternative actions continually to make optimal choices for its survival. Because the reward resulting from a given action can be substantially delayed, the process of linking a reward to its causative action would be facilitated by memory signals related to the animal's previous actions. Although the ventral striatum has been proposed to play a key role in updating the information about the rewards expected from specific actions, it is not known whether the signals related to previous actions exist in the ventral striatum. In the present study, we recorded neuronal ensemble activity in the rat ventral striatum during a visual discrimination task and investigated whether neuronal activity in the ventral striatum encoded signals related to animal's previous actions. The results show that many neurons modulated their activity according to the animal's goal choice in the previous trial, indicating that memory signals for previous actions are available in the ventral striatum. In contrast, few neurons conveyed signals on impending goal choice of the animal, suggesting the absence of decision signals in the ventral striatum. Memory signals for previous actions might contribute to the process of updating the estimates of rewards expected from alternative actions in the ventral striatum.     

Influence of expectation of different rewards on behavior-related neuronal activity in the striatum

This study investigated how different expected rewards influence behavior-related neuronal activity in the anterior striatum. In a spatial delayed-response task, monkeys reached for a left or right target and obtained a small quantity of one of two juices (apple, grenadine, orange, lemon, black currant, or raspberry). In each trial, an initial instruction picture indicated the behavioral target and predicted the reward. Nonmovement trials served as controls for movement relationships. Consistent preferences in special reward choice trials and differences in anticipatory licks, performance errors, and reaction times indicated that animals differentially expected the rewards predicted by the instructions. About 600 of >2,500 neurons in anterior parts of caudate nucleus, putamen, and ventral striatum showed five forms of task-related activations, comprising responses to instructions, spatial or nonspatial activations during the preparation or execution of the movement, and activations preceding or following the rewards. About one-third of the neurons showed different levels of task-related activity depending on which liquid reward was predicted at trial end. Activations were either higher or lower for rewards that were preferred by the animals as compared with nonpreferred rewards. These data suggest that the expectation of an upcoming liquid reward may influence a fraction of task-related neurons in the anterior striatum. Apparently the information about the expected reward is incorporated into the neuronal activity related to the behavioral reaction leading to the reward. The results of this study are in general agreement with an account of goal-directed behavior according to which the outcome should be represented already at the time at which the behavior toward the outcome is performed.     

Neurons in the thalamic CM-Pf complex supply striatal neurons with information about behaviorally significant sensory events

The projection from the thalamic centre médian-parafascicular (CM-Pf) complex to the caudate nucleus and putamen forms a massive striatal input system in primates. We examined the activity of 118 neurons in the CM and 62 neurons in the Pf nuclei of the thalamus and 310 tonically active neurons (TANs) in the striatum in awake behaving macaque monkeys and analyzed the effects of pharmacologic inactivation of the CM-Pf on the sensory responsiveness of the striatal TANs. A large proportion of CM and Pf neurons responded to visual (53%) and/or auditory beep (61%) or click (91%) stimuli presented in behavioral tasks, and many responded to unexpected auditory, visual, or somatosensory stimuli presented outside the task context. The neurons fell into two classes: those having short-latency facilitatory responses (SLF neurons, predominantly in the Pf) and those having long-latency facilitatory responses (LLF neurons, predominantly in the CM). Responses of both types of neuron appeared regardless of whether or not the sensory stimuli were associated with reward. These response characteristics of CM-Pf neurons sharply contrasted with those of TANs in the striatum, which under the same conditions responded preferentially to stimuli associated with reward. Many CM-Pf neurons responded to alerting stimuli such as unexpected handclaps and noises only for the first few times that they occurred; after that, the identical stimuli gradually became ineffective in evoking responses. Habituation of sensory responses was particularly common for the LLF neurons. Inactivation of neuronal activity in the CM and Pf by local infusion of the GABA(A) receptor agonist, muscimol, almost completely abolished the pause and rebound facilitatory responses of TANs in the striatum. Such injections also diminished behavioral responses to stimuli associated with reward. We suggest that neurons in the CM and Pf supply striatal neurons with information about behaviorally significant sensory events that can activate conditional responses of striatal neurons in combination with dopamine-mediated nigrostriatal inputs having motivational value.     

The encoding of cocaine vs. natural rewards in the striatum of nonhuman primates: categories with different activations

The behavioral and motivational changes that result from use of abused substances depend upon activation of neuronal populations in the reward centers of the brain, located primarily in the corpus striatum in primates. To gain insight into the cellular mechanisms through which abused drugs reinforce behavior in the primate brain, changes in firing of neurons in the ventral (VStr, nucleus accumbens) and dorsal (DStr, caudate-putamen) striatum to “natural” (juice) vs. drug (i.v. cocaine) rewards were examined in four rhesus monkeys performing a visual Go-Nogo decision task. Task-related striatal neurons increased firing to one or more of the specific events that occurred within a trial represented by (1) Target stimuli (Go trials) or (2) Nogotarget stimuli (Nogo trials), and (3) Reward delivery for correct performance. These three cell populations were further subdivided into categories that reflected firing exclusively on one or the other type of signaled reward (juice or cocaine) trial (20%–30% of all cells), or, a second subpopulation that fired on both (cocaine and juice) types of rewarded trial (50%). Results show that neurons in the primate striatum encoded cocaine-rewarded trials similar to juice-rewarded trials, except for (1) increased firing on cocaine-rewarded trials, (2) prolonged activation during delivery of i.v. cocaine infusion, and (3) differential firing in ventral (VStr cells) vs. dorsal (DStr cells) striatum cocaine-rewarded trials. Reciprocal activations of antithetic subpopulations of cells during different temporal intervals within the same trial suggest a functional interaction between processes that encode drug and natural rewards in the primate brain.     

Responses of tonically discharging neurons in the monkey striatum to primary rewards delivered during different behavioral states

In the primate striatum, the tonically discharging neurons respond to conditioned stimuli associated with reward. We investigated whether these neurons respond to the reward itself and how changes in the behavioral context in which the reward is delivered might influence their responsiveness. A total of 286 neurons in the caudate nucleus and putamen were studied in two awake macaque monkeys while liquid reward was delivered in three behavioral situations: (1) an instrumental task, in which reward was delivered upon execution of a visually triggered arm movement; (2) a classically conditioned task, in which reward was delivered 1 s after a visual signal; (3) a free reward situation, in which reward was delivered at irregular time intervals outside of any conditioning task. The monkeys′ uncertainty about the time at which reward will be delivered was assessed by monitoring their mouth movements. A larger proportion of neurons responsive to reward was observed in the free reward situation (86%) than in the classically conditioned (57%) and instrumental tasks (37%). Among the neurons tested in all situations (n = 78), 24% responded to reward regardless of the situation and 65% in only one or two situations. Responses selective for one particular situation occurred exclusively in the free reward situation. When the reward was delivered immediately after the visual signal in the classically conditioned task, most of the neurons reduced or completely lost their responses to reward, and other neurons remained responsive. Conversely, neuronal responses invariably persisted when reward was delivered later than 1 s after the visual signal. This is the first report that tonic striatal neurons might display responses directly to primary rewards. The neuronal responses were strongly influenced by the behavioral context in which the animals received the reward. An important factor appears to be the timing of reward. These neurons might therefore contribute to a general aspect of behavioral reactivity of the subject to relevant stimuli. Received: 16 September 1996 / Accepted: 1 April 1997     

Neural coding of reward-prediction error signals during classical conditioning with attractive faces

Attractive faces can be considered to be a form of visual reward. Previous imaging studies have reported activity in reward structures including orbitofrontal cortex and nucleus accumbens during presentation of attractive faces. Given that these stimuli appear to act as rewards, we set out to explore whether it was possible to establish conditioning in human subjects by pairing presentation of arbitrary affectively neutral stimuli with subsequent presentation of attractive and unattractive faces. Furthermore, we scanned human subjects with functional magnetic resonance imaging (fMRI) while they underwent this conditioning procedure to determine whether a reward-prediction error signal is engaged during learning with attractive faces as is known to be the case for learning with other types of reward such as juice and money. Subjects showed changes in behavioral ratings to the conditioned stimuli (CS) when comparing post- to preconditioning evaluations, notably for those CSs paired with attractive female faces. We used a simple Rescorla-Wagner learning model to generate a reward-prediction error signal and entered this into a regression analysis with the fMRI data. We found significant prediction error-related activity in the ventral striatum during conditioning with attractive compared with unattractive faces. These findings suggest that an arbitrary stimulus can acquire conditioned value by being paired with pleasant visual stimuli just as with other types of reward such as money or juice. This learning process elicits a reward-prediction error signal in a main target structure of dopamine neurons: the ventral striatum. The findings we describe here may provide insights into the neural mechanisms tapped into by advertisers seeking to influence behavioral preferences by repeatedly exposing consumers to simple associations between products and rewarding visual stimuli such as pretty faces.     

Effects of expectations for different reward magnitudes on neuronal activity in primate striatum

In behavioral science, it is well known that humans and nonhuman animals are highly sensitive to differences in reward magnitude when choosing an outcome from a set of alternatives. We know that a realm of behavioral reactions is altered when animals begin to expect different levels of reward outcome. Our present aim was to investigate how the expectation for different magnitudes of reward influences behavior-related neurophysiology in the anterior striatum. In a spatial delayed response task, different instruction pictures are presented to the monkey. Each image represents a different magnitude of juice. By reaching to the spatial location where an instruction picture was presented, animals could receive the particular liquid amount designated by the stimulus. Reliable preferences in reward choice trials and differences in anticipatory licks, performance errors, and reaction times indicated that animals differentially expected the various reward amounts predicted by the instruction cues. A total of 374 of 2,000 neurons in the anterior parts of the caudate nucleus, putamen, and ventral striatum showed five forms of task-related activation during the preparation or execution of movement and activations preceding or following the liquid drop delivery. Approximately one-half of these striatal neurons showed differing response levels dependent on the magnitude of liquid to be received. Results of a linear regression analysis showed that reward magnitude and single cell discharge rate were related in a subset of neurons by a monotonic positive or negative relationship. Overall, these data support the idea that the striatum utilizes expectancies that contain precise information concerning the predicted, forthcoming level of reward in directing general behavioral reactions.     

Neuronal activity in primate orbitofrontal cortex reflects the value of time

Neurons in monkey orbitofrontal cortex (OF) are known to respond to reward-predicting cues with a strength that depends on the value of the predicted reward as determined 1) by intrinsic attributes including size and quality and 2) by extrinsic factors including the monkey's state of satiation and awareness of what other rewards are currently available. We pose here the question whether another extrinsic factor critical to determining reward value-the delay expected to elapse before delivery-influences neuronal activity in OF. To answer this question, we recorded from OF neurons while monkeys performed a memory-guided saccade task in which a cue presented early in each trial predicted whether the delay before the monkey could respond and receive a reward of fixed size would be short or long. OF neurons tended to fire more strongly in response to a cue predicting a short delay. The tendency to fire more strongly in anticipation of a short delay was correlated across neurons with the tendency to fire more strongly before a large reward. We conclude that neuronal activity in OF represents the time-discounted value of the expected reward.     

Postsynaptic potentials and axonal projections of tegmental neurons responding to electrical stimulation of the toad striatum

The amphibian telencephalic striatum as a major component of the basal ganglia receives multisensory information and projects to the tegmentum and other structures. However, how striatal neurons modulate tegmental activity remains unknown. Here, we show by using intracellular recording and staining in toads that electrical stimulation of the ipsilateral striatum evoked an inhibitory postsynaptic potential (IPSP) in presumably binocular tegmental neurons. Seventy-one neurons were intracellularly stained with Lucifer yellow or horseradish peroxidase. They were located in the anterodorsal tegmental nucleus, anteroventral tegmental nucleus, nucleus profundus mesencephali, and superficial isthmal reticular nucleus, with axons projecting to the tectum, nucleus isthmi, and spinal cord. It appears that the striatum can control visually guided behaviors through the striato-tegmento-spinal pathway and the tegmento-spinal pathway mediated by the tectum and nucleus isthmi.     

Enhanced c-Fos expression in superior colliculus,paraventricular thalamus and septum during learning of cue-reward association

Reward-mediated associative learning is important for recognizing the significance of environmental cues. Such learning involves convergence of multimodal sensory inputs with circuits involved in affective and memory processes. Dopamine-dependent plasticity in the striatum plays a pivotal role, but the wider circuits engaged in cue-reward association are poorly understood. To identify candidate structures that may be of particular interest for further detailed electrophysiological and functional analysis, we quantified c-Fos expression in a selection of brain structures. c-Fos is a well-known marker of cell activation with additional potential importance for synaptic plasticity. We compared c-Fos expression between animals exposed to 100 pairings of a novel conditioned stimulus with a subsequent reward, and control animals exposed to the same number of cues and rewards, but where the cues and rewards occurred at random with respect to each other. We found significant increases in c-Fos expression in the superior colliculus in the group exposed to cue-reward pairing. This is consistent with previous recordings in conscious animals, showing modulation of phasic visual responses of single collicular neurons depending on their association with reward. Further, the data also suggest the possibility that the thalamic paraventricular nucleus and septal nuclei may be selectively activated during cue-reward association learning. Little is known of the neurophysiological responses in these structures during such tasks, so the present results suggest they would be targets of interest for future single-neuron recording experiments, designed to confirm whether the neurons show learning-specific modulation.     

Neurons in the macaque orbitofrontal cortex code relative preference of both rewarding and aversive outcomes

Many studies have shown that the orbitofrontal cortex (OFC) is involved in the processing of emotional information. However, although some lines of study showed that the OFC is also involved in negative emotions, few electrophysiological studies have focused on the characteristics of OFC neuronal responses to aversive information at the individual neuron level. On the other hand, a previous study has shown that many OFC neurons code relative preference of available rewards. In this study, we aimed to elucidate how reward information and aversive information are coded in the OFC at the individual neuron level. To achieve this aim, we introduced the electrical stimulus (ES) as an aversive stimulus, and compared the neuronal responses to the ES-predicting stimulus with those to reward-predicting stimuli. We found that many OFC neurons showed responses to both the ES-predicting stimulus and the reward-predicting stimulus, and they code relative preference of not only the reward outcome but also the aversive outcome. This result suggests that the same group of OFC neurons code both reward and aversive information in the form of relative preference.     

Weekend–weekday advances in sleep timing are associated with altered reward-related brain function in healthy adolescents

Sleep timing shifts later during adolescence, thus conflicting with early school start times. This can lead to irregular weekday–weekend schedules and circadian misalignment, which have been linked to depression and substance abuse, consistent with disruptions in the processing of rewards. We tested associations between weekend–weekday shifts in sleep timing and the neural response to monetary reward in healthy adolescents, using actigraphy and a functional magnetic resonance imaging paradigm. Region-of-interest analyses focused on the medial prefrontal cortex (mPFC) and striatum, both of which are implicated in reward function. Analyses adjusted for pubertal stage, sex, and total sleep time. Greater weekend–weekday advances in midsleep were associated with decreased mPFC and striatal reactivity to reward, which could reflect reduced regulatory response and reward sensitivity. We speculate that circadian misalignment associated with weekend shifts in sleep timing may contribute to reward-related problems such as depression and substance abuse.     

Weekend–weekday advances in sleep timing are associated with altered reward-related brain function in healthy adolescents

Sleep timing shifts later during adolescence, thus conflicting with early school start times. This can lead to irregular weekday–weekend schedules and circadian misalignment, which have been linked to depression and substance abuse, consistent with disruptions in the processing of rewards. We tested associations between weekend–weekday shifts in sleep timing and the neural response to monetary reward in healthy adolescents, using actigraphy and a functional magnetic resonance imaging paradigm. Region-of-interest analyses focused on the medial prefrontal cortex (mPFC) and striatum, both of which are implicated in reward function. Analyses adjusted for pubertal stage, sex, and total sleep time. Greater weekend–weekday advances in midsleep were associated with decreased mPFC and striatal reactivity to reward, which could reflect reduced regulatory response and reward sensitivity. We speculate that circadian misalignment associated with weekend shifts in sleep timing may contribute to reward-related problems such as depression and substance abuse.     

Organization of ventral tegmental area projections to the ventral tegmental area-nigral complex in the rat

The ventral tegmental area (VTA) is a nodal link in reward circuitry. Based on its striatal output, it has been subdivided in a caudomedial part which targets the ventromedial striatum, and a lateral part which targets the ventrolateral striatum [Ikemoto S (2007) Dopamine reward circuitry: two projection systems from the ventral midbrain to the nucleus accumbens-olfactory tubercle complex. Brain Res Rev 56:27-78]. Whether these two VTA parts are interconnected and to what extent the VTA innervates the substantia nigra compacta (SNc) and retrorubral nucleus (RR) are critical issues for understanding information processing in the basal ganglia. Here, VTA projections to the VTA-nigral complex were examined in rats, using Phaseolus vulgaris leucoagglutinin (PHA-L) as anterograde tracer. The results show that the dorsolateral VTA projects to itself, as well as to the dorsal tier of the SNc and RR, largely avoiding the caudomedial VTA. The ventrolateral VTA innervates mainly the interfascicular nucleus. The components of the caudomedial VTA (the interfascicular, paranigral and caudal linear nuclei) are connected with each other. In addition, the caudomedial VTA (especially the paranigral and caudal linear nuclei) innervates the lateral VTA, and, to a lesser degree, the SNc and RR. The caudal pole of the VTA sends robust, bilateral projections to virtually all the VTA-nigral complex, which terminate in the dorsal and ventral tiers. Modest inputs from the medial supramammillary nucleus to ventromedial parts of the VTA-nigral complex were also identified. In double-immunostained sections, PHA-L-labeled varicosities were sometimes found apposed to tyrosine hydroxylase-positive neurons in the ventral mesencephalon. Overall, the results underscore that VTA projections to the VTA-nigral complex are substantial and topically organized. In general, these projections, like the spiralated striato-nigro-striatal loops, display a medial-to-lateral organization. This anatomical arrangement conceivably permits the ventromedial striatum to influence the activity of the lateral striatum. The caudal pole of the VTA appears to be a critical site for a global recruitment of the mesotelencephalic system.     

Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops

Evaluation of both immediate and future outcomes of one's actions is a critical requirement for intelligent behavior. Using functional magnetic resonance imaging (fMRI), we investigated brain mechanisms for reward prediction at different time scales in a Markov decision task. When human subjects learned actions on the basis of immediate rewards, significant activity was seen in the lateral orbitofrontal cortex and the striatum. When subjects learned to act in order to obtain large future rewards while incurring small immediate losses, the dorsolateral prefrontal cortex, inferior parietal cortex, dorsal raphe nucleus and cerebellum were also activated. Computational model-based regression analysis using the predicted future rewards and prediction errors estimated from subjects' performance data revealed graded maps of time scale within the insula and the striatum: ventroanterior regions were involved in predicting immediate rewards and dorsoposterior regions were involved in predicting future rewards. These results suggest differential involvement of the cortico-basal ganglia loops in reward prediction at different time scales.     

Effects of acute dopamine depletion on the electrophysiological properties of striatal neurons

The striatum, the main input nucleus of basal ganglia, receives a massive innervation from the entire cerebral cortex and is in charge of the detection of behaviorally relevant signals. In turn, via its projections to the output nuclei of basal ganglia, the striatum contributes to the organization of appropriate compartmental responses. Substantia nigra pars compacta dopaminergic neurons project predominantly to the striatum and regulate striatal functions. Implications of dopaminergic receptors on the physiology of striatal neurons are now well documented. By contrast, the effects of acute dopamine depletion on striatal neurons remain poorly explored. Here, the alpha-methyl-para-tyrosine was used to deplete dopamine from rat brain slices. We analyzed the consequences of a alpha-methyl-para-tyrosine treatment on membrane properties of striatal neurons: the medium-sized spiny neurons and the interneurons (GABAergic, cholinergic and NO-synthase). After acute dopamine depletion, medium-sized spiny neurons became more excitable. GABAergic interneurons became less excitable whereas cholinergic cells displayed an increased excitability. NO-synthase-containing interneurons did not show noticeable changes in their excitability. Such membrane properties changes indicate that striatal circuits should undergo major alteration in cortico-basal ganglia information processing.     

Complex response to afferent excitatory bursts by nucleus accumbens medium spiny projection neurons

The nucleus accumbens (NAc) of the ventral striatum is involved in attention, motivation, movement, learning, reward, and addiction. GABAergic medium spiny projection neurons that make up approximately 90% of the neuronal population are commonly driven by convergent bursts of afferent excitation. We monitored spiny projection neurons in mouse striatal slices while applying stimulus trains to mimic bursts of excitation. A stimulus train evoked a simple, short-lived postsynaptic response from CA1 hippocampal pyramidal neurons, but the train evoked a complex series of excitatory postsynaptic potentials (EPSPs) or currents (EPSCs) from the NAc spiny projection neurons. As is commonly seen with projection neurons, the EPSC amplitudes initially displayed facilitation followed by depression, and that pattern was sensitive to the extracellular calcium concentration. In addition, there were two other novel observations. The spiny projection neurons responded to the stimulus train with a prolonged depolarization that was accompanied by a posttrain increase of spontaneous glutamatergic synaptic activity. Blocking AMPA/kainate glutamate receptors strongly inhibited the evoked EPSP/EPSCs, the posttrain spontaneous synaptic activity, and the prolonged depolarization. A potassium channel inhibitor increased and extended the prolonged postsynaptic depolarization, causing a long-lasting depolarized plateau potential. Our results indicate that burst-like activity along ventral striatal afferents is extended in time by additional spontaneous glutamate release that is integrated by the postsynaptic spiny projection neurons into a prolonged depolarization. The results suggest that the posttrain quantal glutamate release helps to blend and maintain multiple afferent inputs. That convergent excitation is further integrated by the postsynaptic neuron into a prolonged depolarization that may contribute to the depolarized "up state" observed in vivo.