GRP78 counteracts cell death and protein aggregation caused by mutant huntingtin proteins

Pain and reward are opponent, interacting processes. Such interactions are enabled by neuroanatomical and neurochemical overlaps of brain systems that process pain and reward. Cerebral processing of hedonic ('liking') and motivational ('wanting') aspects of reward can be separated: the orbitofrontal cortex and opioids play an important role for the hedonic experience, and the ventral striatum and dopamine predominantly process motivation for reward. Supported by neuroimaging studies, we present here the hypothesis that the orbitofrontal cortex and opioids are responsible for pain modulation by hedonic experience, while the ventral striatum and dopamine mediate motivational effects on pain. A rewarding stimulus that appears to be particularly important in the context of pain is pain relief. Further, reward, including pain relief, leads to operant learning, which can affect pain sensitivity. Indirect evidence points at brain mechanisms that might underlie pain relief as a reward and related operant learning but studies are scarce. Investigating the cerebral systems underlying pain-reward interactions as well as related operant learning holds the potential of better understanding mechanisms that contribute to the development and maintenance of chronic pain, as detailed in the last section of this review.     

A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task.

This study investigated how the simulated response of dopamine neurons to reward-related stimuli could be used as reinforcement signal for learning a spatial delayed response task. Spatial delayed response tasks assess the functions of frontal cortex and basal ganglia in short-term memory, movement preparation and expectation of environmental events. In these tasks, a stimulus appears for a short period at a particular location, and after a delay the subject moves to the location indicated. Dopamine neurons are activated by unpredicted rewards and reward-predicting stimuli, are not influenced by fully predicted rewards, and are depressed by omitted rewards. Thus, they appear to report an error in the prediction of reward, which is the crucial reinforcement term in formal learning theories. Theoretical studies on reinforcement learning have shown that signals similar to dopamine responses can be used as effective teaching signals for learning. A neural network model implementing the temporal difference algorithm was trained to perform a simulated spatial delayed response task. The reinforcement signal was modeled according to the basic characteristics of dopamine responses to novel stimuli, primary rewards and reward-predicting stimuli. A Critic component analogous to dopamine neurons computed a temporal error in the prediction of reinforcement and emitted this signal to an Actor component which mediated the behavioral output. The spatial delayed response task was learned via two subtasks introducing spatial choices and temporal delays, in the same manner as monkeys in the laboratory. In all three tasks, the reinforcement signal of the Critic developed in a similar manner to the responses of natural dopamine neurons in comparable learning situations, and the learning curves of the Actor replicated the progress of learning observed in the animals. Several manipulations demonstrated further the efficacy of the particular characteristics of the dopamine-like reinforcement signal. Omission of reward induced a phasic reduction of the reinforcement signal at the time of the reward and led to extinction of learned actions. A reinforcement signal without prediction error resulted in impaired learning because of perseverative errors. Loss of learned behavior was seen with sustained reductions of the reinforcement signal, a situation in general comparable to the loss of dopamine innervation in Parkinsonian patients and experimentally lesioned animals. The striking similarities in teaching signals and learning behavior between the computational and biological results suggest that dopamine-like reward responses may serve as effective teaching signals for learning behavioral tasks that are typical for primate cognitive behavior, such as spatial delayed responding.     

Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex

The orbitofrontal cortex has been hypothesized to carry information regarding the value of expected rewards. Such information is essential for associative learning, which relies on comparisons between expected and obtained reward for generating instructive error signals. These error signals are thought to be conveyed by dopamine neurons. To test whether orbitofrontal cortex contributes to these error signals, we recorded from dopamine neurons in orbitofrontal-lesioned rats performing a reward learning task. Lesions caused marked changes in dopaminergic error signaling. However, the effect of lesions was not consistent with a simple loss of information regarding expected value. Instead, without orbitofrontal input, dopaminergic error signals failed to reflect internal information about the impending response that distinguished externally similar states leading to differently valued future rewards. These results are consistent with current conceptualizations of orbitofrontal cortex as supporting model-based behavior and suggest an unexpected role for this information in dopaminergic error signaling.     

Neural coding of reward-prediction error signals during classical conditioning with attractive faces

Attractive faces can be considered to be a form of visual reward. Previous imaging studies have reported activity in reward structures including orbitofrontal cortex and nucleus accumbens during presentation of attractive faces. Given that these stimuli appear to act as rewards, we set out to explore whether it was possible to establish conditioning in human subjects by pairing presentation of arbitrary affectively neutral stimuli with subsequent presentation of attractive and unattractive faces. Furthermore, we scanned human subjects with functional magnetic resonance imaging (fMRI) while they underwent this conditioning procedure to determine whether a reward-prediction error signal is engaged during learning with attractive faces as is known to be the case for learning with other types of reward such as juice and money. Subjects showed changes in behavioral ratings to the conditioned stimuli (CS) when comparing post- to preconditioning evaluations, notably for those CSs paired with attractive female faces. We used a simple Rescorla-Wagner learning model to generate a reward-prediction error signal and entered this into a regression analysis with the fMRI data. We found significant prediction error-related activity in the ventral striatum during conditioning with attractive compared with unattractive faces. These findings suggest that an arbitrary stimulus can acquire conditioned value by being paired with pleasant visual stimuli just as with other types of reward such as money or juice. This learning process elicits a reward-prediction error signal in a main target structure of dopamine neurons: the ventral striatum. The findings we describe here may provide insights into the neural mechanisms tapped into by advertisers seeking to influence behavioral preferences by repeatedly exposing consumers to simple associations between products and rewarding visual stimuli such as pretty faces.     

A reservoir of time constants for memory traces in cortical neurons

According to reinforcement learning theory of decision making, reward expectation is computed by integrating past rewards with a fixed timescale. In contrast, we found that a wide range of time constants is available across cortical neurons recorded from monkeys performing a competitive game task. By recognizing that reward modulates neural activity multiplicatively, we found that one or two time constants of reward memory can be extracted for each neuron in prefrontal, cingulate and parietal cortex. These timescales ranged from hundreds of milliseconds to tens of seconds, according to a power law distribution, which is consistent across areas and reproduced by a 'reservoir' neural network model. These neuronal memory timescales were weakly, but significantly, correlated with those of monkey's decisions. Our findings suggest a flexible memory system in which neural subpopulations with distinct sets of long or short memory timescales may be selectively deployed according to the task demands.     

Convergence of sensory systems in the orbitofrontal cortex in primates and brain design for emotion

In primates, stimuli to sensory systems influence motivational and emotional behavior via neural relays to the orbitofrontal cortex. This article reviews studies on the effects of stimuli from multiple sensory modalities on the brain of humans and some other higher primates. The primate orbitofrontal cortex contains the secondary taste cortex, in which the reward value of taste is represented. It also contains the secondary and tertiary olfactory cortical areas, in which information about the identity and also about the reward value of odors is represented. A somatosensory input is revealed by neurons that respond to the viscosity of food in the mouth, to the texture (mouth feel) of fat in the mouth, and to the temperature of liquids placed into the mouth. The orbitofrontal cortex also receives information about the sight of objects from the temporal lobe cortical visual areas. Information about each of these modalities is represented separately by different neurons, but in addition, other neurons show convergence between different types of sensory input. This convergence occurs by associative learning between the visual or olfactory input and the taste. In that emotions can be defined as states elicited by reinforcers, the neurons that respond to primary reinforcers (such as taste and touch), as well as learn associations to visual and olfactory stimuli that become secondary reinforcers, provide a basis for understanding the functions of the orbitofrontal cortex in emotion. In complementary neuroimaging studies in humans, it is being found that areas of the orbitofrontal cortex are activated by pleasant touch, by painful touch, by taste, by smell, and by more abstract reinforcers such as winning or losing money. Damage to the orbitofrontal cortex in humans can impair the learning and reversal of stimulus-reinforcement associations and thus the correction of behavioral responses when these are no longer appropriate because previous reinforcement contingencies change. It is striking that humans and other catarrhines, being visual specialists like other anthropoids, interface the visual system to other sensory systems (e.g., taste and smell) in the orbitofrontal cortex.     

The orbitofrontal cortex and beyond: from affect to decision-making

The orbitofrontal cortex represents the reward or affective value of primary reinforcers including taste, touch, texture, and face expression. It learns to associate other stimuli with these to produce representations of the expected reward value for visual, auditory, and abstract stimuli including monetary reward value. The orbitofrontal cortex thus plays a key role in emotion, by representing the goals for action. The learning process is stimulus-reinforcer association learning. Negative reward prediction error neurons are related to this affective learning. Activations in the orbitofrontal cortex correlate with the subjective emotional experience of affective stimuli, and damage to the orbitofrontal cortex impairs emotion-related learning, emotional behaviour, and subjective affective state. With an origin from beyond the orbitofrontal cortex, top-down attention to affect modulates orbitofrontal cortex representations, and attention to intensity modulates representations in earlier cortical areas of the physical properties of stimuli. Top-down word-level cognitive inputs can bias affective representations in the orbitofrontal cortex, providing a mechanism for cognition to influence emotion. Whereas the orbitofrontal cortex provides a representation of reward or affective value on a continuous scale, areas beyond the orbitofrontal cortex such as the medial prefrontal cortex area 10 are involved in binary decision-making when a choice must be made. For this decision-making, the orbitofrontal cortex provides a representation of each specific reward in a common currency.     

Role of rodent secondary motor cortex in value-based action selection

Despite widespread neural activity related to reward values, signals related to upcoming choice have not been clearly identified in the rodent brain. Here we examined neuronal activity in the lateral (AGl) and medial (AGm) agranular cortex, corresponding to the primary and secondary motor cortex, respectively, in rats performing a dynamic foraging task. Choice signals, before behavioral manifestation of the rat's choice, arose in the AGm earlier than in any other areas of the rat brain previously studied under free-choice conditions. The AGm also conveyed neural signals for decision value and chosen value. By contrast, upcoming choice signals arose later, and value signals were weaker, in the AGl. We also found that AGm lesions made the rats' choices less dependent on dynamically updated values. These results suggest that rodent secondary motor cortex might be uniquely involved in both representing and reading out value signals for flexible action selection.     

Learning of sequential movements by neural network model with dopamine-like reinforcement signal

Dopamine neurons appear to code an error in the prediction of reward. They are activated by unpredicted rewards, are not influenced by predicted rewards, and are depressed when a predicted reward is omitted. After conditioning, they respond to reward-predicting stimuli in a similar manner. With these characteristics, the dopamine response strongly resembles the predictive reinforcement teaching signal of neural network models implementing the temporal difference learning algorithm. This study explored a neural network model that used a reward-prediction error signal strongly resembling dopamine responses for learning movement sequences. A different stimulus was presented in each step of the sequence and required a different movement reaction, and reward occurred at the end of the correctly performed sequence. The dopamine-like predictive reinforcement signal efficiently allowed the model to learn long sequences. By contrast, learning with an unconditional reinforcement signal required synaptic eligibility traces of longer and biologically less-plausible durations for obtaining satisfactory performance. Thus, dopamine-like neuronal signals constitute excellent teaching signals for learning sequential behavior.     

Instructive signals for motor learning from visual cortical area MT

Sensory error signals have long been proposed to act as instructive signals to guide motor learning. Here we have exploited the temporal specificity of learning in smooth pursuit eye movements and the well-defined anatomical structure of the neural circuit for pursuit to identify a part of sensory cortex that provides instructive signals for motor learning in monkeys. We show that electrical microstimulation in the motion-sensitive middle temporal area (MT) of extrastriate visual cortex instructs learning in smooth eye movements in a way that closely mimics the learning instructed by real visual motion. We conclude that MT provides instructive signals for motor learning in smooth pursuit eye movements under natural conditions, suggesting a similar role for sensory cortices in many kinds of learned behaviors.     

Olfactory learning induces differential long-lasting changes in rat central olfactory pathways

In the present work, we investigated lasting changes induced by olfactory learning at different levels of the olfactory pathways. For this, evoked field potentials induced by electrical stimulation of the olfactory bulb were recorded simultaneously in the anterior piriform cortex, the posterior piriform cortex, the lateral entorhinal cortex and the dentate gyrus. The amplitude of the evoked field potential's main component was measured in each site before, immediately after, and 20 days after completion of associative learning. Evoked field potential recordings were carried out under two experimental conditions in the same animals: awake and anesthetized. In the learning task, rats were trained to associate electrical stimulation of one olfactory bulb electrode with the delivery of sucrose (positive reward), and stimulation of a second olfactory bulb electrode with the delivery of quinine (negative reward). In this way, stimulation of the same olfactory bulb electrodes used for inducing field potentials served as a discriminative cue in the learning paradigm. The data showed that positively reinforced learning resulted in a lasting increase in evoked field potential amplitude restricted to posterior piriform cortex and lateral entorhinal cortex. In contrast, negatively reinforced learning was mainly accompanied by a decrease in evoked field potential amplitude in the dentate gyrus. Moreover, the expression of these learning-related changes occurred to be modulated by the animals arousal state. Indeed, the comparison between anesthetized versus awake animals showed that although globally similar, the changes were expressed earlier with respect to learning, under anesthesia than in the awake state. From these data we suggest that associative olfactory learning involves different neural circuits depending on the acquired value of the stimulus. Furthermore, they show the existence of a functional dissociation between anterior and posterior piriform cortex in mnesic processes, and stress the importance of the animal's arousal state on the expression of learning-induced plasticity.     

Cue and reward signals carried by monkey entorhinal cortex neurons during reward schedules

Ablation of entorhinal/perirhinal cortices prevents learning associations between visual stimuli used as cues in reward schedules and the schedule state. Single neurons in perirhinal cortex are sensitive to associations between the cues and the reward schedules. To investigate whether neurons in the entorhinal cortex have similar sensitivities, we recorded single neuronal activity from two rhesus monkeys while the monkeys performed a visually cued reward schedule task. When the cue was related to the reward schedules, the monkeys made progressively fewer errors as the schedule state became closer to the reward state, showing that the monkeys were sensitive to the cue and the schedule state. Of 75 neurons recorded in the entorhinal cortex during task performance, about 30% responded. About half of these responded after cue presentation. When the relation of the cue to the reward schedules was random, the cue-related responses disappeared or lost their selectivity for schedule states. The responses of the entorhinal cortex neurons are similar to responses of perirhinal cortex neurons in that they are selective for the associative relationships between cues and reward schedules. However, they are particularly selective for the first trial of a new schedule, in contrast to perirhinal cortex where responsivity to all schedule states is seen. A different subpopulation of entorhinal neurons responded to the reward, unlike perirhinal neurons which respond solely to the cue. These results indicate that the entorhinal signals carry associative relationships between the visual cues and reward schedules, and between rewards and reward schedules that are not simply derived from perirhinal cortex by feed-forward serial processing.     

Smokers' brains compute, but ignore, a fictive error signal in a sequential investment task

Addicted individuals pursue substances of abuse even in the clear presence of positive outcomes that may be foregone and negative outcomes that may occur. Computational models of addiction depict the addicted state as a feature of a valuation disease, where drug-induced reward prediction error signals steer decisions toward continued drug use. Related models admit the possibility that valuation and choice are also directed by 'fictive' outcomes (outcomes that have not been experienced) that possess their own detectable error signals. We hypothesize that, in addiction, anomalies in these fictive error signals contribute to the diminished influence of potential consequences. Using a simple investment game and functional magnetic resonance imaging in chronic cigarette smokers, we measured neural and behavioral responses to error signals derived from actual experience and from fictive outcomes. In nonsmokers, both fictive and experiential error signals predicted subjects' choices and possessed distinct neural correlates. In chronic smokers, choices were not guided by error signals derived from what might have happened, despite ongoing and robust neural correlates of these fictive errors. These data provide human neuroimaging support for computational models of addiction and suggest the addition of fictive learning signals to reinforcement learning accounts of drug dependence.     

Dissociated roles of the anterior cingulate cortex in reward and conflict processing as revealed by the feedback error-related negativity and N200

The reinforcement learning theory of the error-related negativity (ERN) holds that the impact of reward signals carried by the midbrain dopamine system modulates activity of the anterior cingulate cortex (ACC), alternatively disinhibiting and inhibiting the ACC following unpredicted error and reward events, respectively. According to a recent formulation of the theory, activity that is intrinsic to the ACC produces a component of the event-related brain potential (ERP) called the N200, and following unpredicted rewards, the N200 is suppressed by extrinsically applied positive dopamine reward signals, resulting in an ERP component called the feedback-ERN (fERN). Here we demonstrate that, despite extensive spatial and temporal overlap between the two ERP components, the functional processes indexed by the N200 (conflict) and the fERN (reward) are dissociable. These results point toward avenues for future investigation.     

TRPs and pain

Nociception is the process of transmission of painful signals by nociceptors in the primary afferent nerve fibers, which specifically respond to noxious stimuli. These noxious stimuli are detected by nociceptors and converted into electrical signals, which are then transmitted to the spinal cord, thalamus, and the cerebral cortex, where pain is finally sensed. Transient receptor potential (TRP) ion channels have emerged as a family of evolutionarily conserved ligand-gated ion channels that function as molecular detectors of physical stimuli. Several member of this family, at least six channels from three TRP family subtypes (TRPV1–4, TRPM8, and TRPA1), are expressed in nociceptors, where they act as transducers for signals from thermal, chemical, and mechanical stimuli and play crucial roles in the generation and development of pathological pain perception. This review focuses on the increasing evidence of TRP channel involvement and contribution in nociceptive pain and the pain hypersensitivity associated with peripheral inflammation or neuropathy, and on the renewed interest in targeting TRP channels for pain relief.     

Risk-sensitive neurons in macaque posterior cingulate cortex

People and animals often demonstrate strong attraction or aversion to options with uncertain or risky rewards, yet the neural substrate of subjective risk preferences has rarely been investigated. Here we show that monkeys systematically preferred the risky target in a visual gambling task in which they chose between two targets offering the same mean reward but differing in reward uncertainty. Neuronal activity in posterior cingulate cortex (CGp), a brain area linked to visual orienting and reward processing, increased when monkeys made risky choices and scaled with the degree of risk. CGp activation was better predicted by the subjective salience of a chosen target than by its actual value. These data suggest that CGp signals the subjective preferences that guide visual orienting.     

Supplementary motor area encodes reward expectancy in eye-movement tasks

Neural activity signifying the expectation of reward has been found recently in many parts of the brain, including midbrain and cortical structures. These signals can facilitate goal-directed behavior or the learning of new skills based on reinforcements. Here we show that neurons in the supplementary motor area (SMA), an area concerned with movements of the body and limbs, also carry a reward expectancy signal in the postsaccadic period of oculomotor tasks. While the monkeys performed blocks of memory-guided and object-based saccades, the neurons discharged a burst after a approximately 200-ms delay following the target-acquiring saccade in the memory task but often fired concurrently with the target-acquiring saccade in the object task. The hypothesis that this postsaccadic bursting activity reflects the expectation of a reward was tested with a series of manipulations to the memory-guided saccade task. It was found that although the timing of the bursting activity corresponds to a visual feedback stimulus, the visual feedback is not required for the neurons to discharge a burst. Second, blocks of no-reward trials reveal an extinction of the bursting activity as the monkeys come to understand that they would not be rewarded for properly generated saccades. Finally, the delivery of unexpected rewards confirmed that in many of the neurons, the activity is not related to a motor plan to acquire the reward (e.g., licking). Thus we conclude that reward expectancy is represented by the activity of SMA neurons, even in the context of an oculomotor task. These results suggest that the reward expectancy signal is broadcast over a large extent of motor cortex, and may facilitate the learning of new, coordinated behavior between different body parts.     

Prefrontal cortex and decision making in a mixed-strategy game

In a multi-agent environment, where the outcomes of one's actions change dynamically because they are related to the behavior of other beings, it becomes difficult to make an optimal decision about how to act. Although game theory provides normative solutions for decision making in groups, how such decision-making strategies are altered by experience is poorly understood. These adaptive processes might resemble reinforcement learning algorithms, which provide a general framework for finding optimal strategies in a dynamic environment. Here we investigated the role of prefrontal cortex (PFC) in dynamic decision making in monkeys. As in reinforcement learning, the animal's choice during a competitive game was biased by its choice and reward history, as well as by the strategies of its opponent. Furthermore, neurons in the dorsolateral prefrontal cortex (DLPFC) encoded the animal's past decisions and payoffs, as well as the conjunction between the two, providing signals necessary to update the estimates of expected reward. Thus, PFC might have a key role in optimizing decision-making strategies.     

From anticipation to action, the role of dopamine in perceptual decision making: an fMRI-tyrosine depletion study

During simple sensorimotor decision making, neurons in the parietal cortex extract evidence from sensory information provided by visual areas until a decision is reached. Contextual information can bias parietal activity during the task and change the decision-making parameters. One type of contextual information is the availability of reward for correct decisions. We tested the hypothesis that the frontal lobes and basal ganglia use contextual information to bias decision making to maximize reward. Human volunteers underwent functional MRI while making decisions about the motion of dots on a computer monitor. On rewarded trials, subjects responded more slowly by increasing the threshold to decision. Rewarded trials were associated with activation in the ventral striatum and prefrontal cortex in the period preceding coherent dot motion, and the degree of activation predicted the increased decision threshold. Decreasing dopamine transmission, using a tyrosine-depleting amino acid mixture, abolished the reward-related corticostriatal activation and eliminated the correlation between striatal activity and decision threshold. These observations provide direct evidence that some reward-related functional MRI signals in the striatum are the result of dopamine neuron activity and demonstrate that mesolimbic dopamine transmission can influence perceptual and decision-making neural processes engaged to maximize reward harvest.     

Choice, uncertainty and value in prefrontal and cingulate cortex

Reinforcement learning models that focus on the striatum and dopamine can predict the choices of animals and people. Representations of reward expectation and of reward prediction errors that are pertinent to decision making, however, are not confined to these regions but are also found in prefrontal and cingulate cortex. Moreover, decisions are not guided solely by the magnitude of the reward that is expected. Uncertainty in the estimate of the reward expectation, the value of information that might be gained by taking a course of action and the cost of an action all influence the manner in which decisions are made through prefrontal and cingulate cortex.