A comprehensive review on the role of testosterone on the neurobehavioral systems implicated in the reinforcement sensitivity theory of personality

ObjectivesThe Reinforcement Sensitivity Theory (RST) is a neuropsychological theory of personality emphasizing approach and avoidance as the two core behavioral aspects. Approach is represented by the behavioral approach system (BAS). Avoidance is represented by the behavioral inhibition system (BIS) and the fight-flight-freeze system (FFFS). Although the influence of testosterone on human behavior has been demonstrated, few studies have investigated the relation between testosterone and the RST. The aim of this narrative review was to decipher the possible role of testosterone on the biological systems involved in the RST in humans.MethodsGoogle scholar, PubMed, PsycARTICLES, PsycINFO, Scopus and Cochrane library databases were interrogated using keywords such as testosterone, BIS, BAS, FFFS, personality, reinforcement sensitivity theory.ResultsSeven original articles, published between 2009 and 2022, assessing the relation between testosterone and the systems implicated in the RST, were included. The results of these studies suggested the presence of a possible positive relation between testosterone and the BAS. However, the impact of testosterone on the BIS and/or FFFS seems to be less clear.DiscussionThe consistency in the results supporting the presence of a positive relation between testosterone and the BAS might lead to the consideration of testosterone as a potential correlate in the clinical assessment of several psychopathologies. The inconsistency in the conclusions regarding the impact of testosterone on the BIS and/or the FFFS might be attributed to the different questionnaires used as measurement tools. Additional research remains needed.     

Neural substrates of opiate reinforcement

1. 1. A major component of opiate reward is derived from a drug action in the ventral tegmental area: (a) rats quickly learn to self-administer morphine directly into the ventral tegmentum, (b) intracranial self-administration into other brain sites is not quickly learned, and (c) narcotic antagonist microinjections into the ventral tegmentum attenuate reward from intravenous heroin infusions.

2. 2. At least one component of opiate reward is dependent on a dopaminergic system: (a) electrophysiological and neurochemical indices suggest that opiates activate ventral tegmental dopaminergic neurons, (b) ventral tegmental opiate infusions are behaviorally activating producing contralateral rotation that is blocked by neuroleptics, (c) reward from heroin is blocked by neuroleptics, and (d) reward from heroin is attenuated by dopamine-depleting lesions of the ventral tegmental system.

3. 3. Brain sites involved in the production of physical dependence on opiates are anatomically distinct from those initiating the acutely rewarding action of opiates.

4. 4. It is theoretically viable that opiates derive their reinforcing impact from a combination of positive and negative reinforcement processes: (a) the neural substrate for the positive reinforcing action probably involves a ventral tegmental dopamine system important in appetitive motivation, and (b) the neural substrate for the negative reinforcing action may involve a periventricular gray system that is independent of the system which mediates the acutely rewarding property of opiates.

Local analysis of behaviour in the adjusting-delay task for assessing choice of delayed reinforcement.

The adjusting-delay task introduced by Mazur (Quantitative analyses of behavior: V. The effect of delay and of intervening events on reinforcement value, 1987, pp. 55-73) has been widely used to study choice of delayed reinforcers. This paradigm involves repeated choice between one reinforcer delivered after a fixed delay and another, typically larger, reinforcer delivered after a variable delay; the variable delay is adjusted depending on the subject's choice until an equilibrium point is reached at which the subject is indifferent between the two alternatives. Rats were trained on a version of this task and their behaviour was examined to determine the nature of their sensitivity to the adjusting delay; these analyses included the use of a cross-correlational technique. No clear evidence of sensitivity to the adjusting delay was found. A number of decision rules, some sensitive to the adjusting delay and some not, were simulated and it was observed that some effects usually supposed to be a consequence of delay sensitivity could be generated by delay-independent processes, such as a consistent, unchanging relative preference between the alternatives. Consequently, the use of explicit analysis of delay sensitivity is advocated in future research on delayed reinforcement.     

Computational perspectives on forebrain microcircuits implicated in reinforcement learning, action selection, and cognitive control

Abundant new information about signaling pathways in forebrain microcircuits presents many challenges, and opportunities for discovery, to computational neuroscientists who strive to bridge from microcircuits to flexible cognition and action. Accurate treatment of microcircuit pathways is especially critical for creating models that correctly predict the outcomes of candidate neurological therapies. Recent models are trying to specify how cortical circuits that enable planning and voluntary actions interact with adaptive subcortical microcircuits in the basal ganglia. The basal ganglia are strongly implicated in reinforcement learning, and in all behavior and cognition over which the frontal lobes exert flexible control. The persisting role of the basal ganglia shows that ancient vertebrate designs for motivated action selection proved adaptable enough to support many “modern” behavioral innovations, including fluent generation of language and speech. This paper summarizes how recent models have incorporated realistic representations of microcircuit features, and have begun to trace their computational implications. Also summarized are recent empirical discoveries that provide guidance regarding how to formulate the rules for synaptic modification that govern learning in cortico-striatal pathways. Such efforts are contributing to an emerging synthesis based on an interlocking set of computational hypotheses regarding cortical interactions with basal ganglia and thalamic nuclei. These hypotheses specify how specialized microcircuits solve learning and control problems inherent to the brain’s parallel design.     

Facilitating tolerance of delayed reinforcement during functional communication training

Few clinical investigations have addressed the problem of delayed reinforcement. In this investigation, three individuals whose destructive behavior was maintained by positive reinforcement were treated using functional communication training (FCT) with extinction (EXT). Next, procedures used in the basic literature on delayed reinforcement and self-control (reinforcer delay fading, punishment of impulsive responding, and provision of an alternative activity during reinforcer delay) were used to teach participants to tolerate delayed reinforcement. With the first case, reinforcer delay fading alone was effective at maintaining low rates of destructive behavior while introducing delayed reinforcement. In the second case, the addition of a punishment component reduced destructive behavior to near-zero levels and facilitated reinforcer delay fading. With the third case, reinforcer delay fading was associated with increases in masturbation and head rolling, but prompting and praising the individual for completing work during the delay interval reduced all problem behaviors and facilitated reinforcer delay fading.     

Immediate and delayed reinforcement on WISC-R performance for mentally retarded students

Thirty white mentally retarded children were administered the WISC-R under one of three conditions: standardized testing conditions, standardized plus immediate reinforcement, standardized plus delayed reinforcement. One reinforcement group earned tokens immediately after each correct response and the other reinforcement group received their earned tokens halfway through and at the end of the IQ test. Both reinforcement groups exchanged their tokens for a variety of backup reinforcers at the halfway point and at the end of the test. Both the immediate and delayed reinforcement groups showed significantly better performance than the standardized testing group, although the two reinforcement groups did not differ from one another. Half of the children in each of the groups that earned tokens scored above the mentally impaired range while only one child in the standardized testing group scored above the mentally impaired level. Implications for the use of intelligence tests for mentally retarded students are discussed.     

On the stability analysis of delayed neural networks systems.

In this paper, the problems of stability of delayed neural networks are investigated, including the stability of discrete and distributed delayed neural networks. Under the generalization of dropping the Lipschitzian hypotheses for output functions, some stability criteria are obtained by using the Liapunov functional method. We do not assume the symmetry of the connection matrix and we establish that the system admits a unique equilibrium point in which the output functions do not satisfy the Lipschitz conditions and do not require them to be differential or strictly monotonously increasing. These criteria can be used to analyze the dynamics of biological neural systems or to design globally stable artificial neural networks.     

Involvement of the rat prefrontal cortex in a delayed reinforcement operant task

To determine whether the rat medial prefrontal cortex (PFC) is involved in delayed reinforcement operant behavior, we studied the effects of transient inactivation of the medial PFC (or the hippocampus as a control) during a delayed reinforcement lever-press task. We demonstrated the involvement of the PFC in this task: PFC inactivation but not hippocampal inactivation significantly impaired performance. In a separate experiment, we also recorded the prefrontal multiple unit activities during the task to determine the roles of the PFC in detail. Neuronal activity decreased during the delay period, suggesting that this decrease plays a role in delayed reinforcement operant behavior.     

Bilateral striatal lesions disrupt performance in an operant delayed reinforcement task in rats

In order to provide an animal model of the impulsivity observed in Huntington's disease, the effects of bilateral neostriatal lesions in rats were evaluated in an operant delayed reinforcement task. When given a choice between responding to one lever for a small but immediate reward and a second lever for a larger delayed reward, normal rats exhibit a marked preference for responding to the high reward lever when the imposed delay is short, but progressively choose the lever associated with immediate small reward as the delays increase. Following striatal lesions, the animals continue to express similar preferences, but the lesions initially impose a distinct flattening of the delay-preference function, suggesting a relative insensitivity to the increasing delay parameter in making their response choices. However, this deficit declines with extend retraining on the task, such that 1-2 months after lesion the delay-dependent shift of preference from the delayed to the immediate lever as the delays lengthened was comparable in lesion and sham animals. Amphetamine further disinhibited all animals, apparent as a further increase in the number and reduction of the latencies of responses made to the lever associated with immediate reward. Striatal lesions had little influence on the effects of amphetamine on task performance, other than the increase in the numbers of omissions of lever and panel responses induced by the drug was more marked in the lesion than sham animals, and the lesioned animals exhibited less delay-dependency than the controls in their preference for responding to the lever associated with the larger delayed reinforcement at the highest (1.5 mg/kg) dose tested. The present results indicate small but clear effects of dorsal striatal lesions in an operant delayed reinforcement task, suggestive of an initial impairment in response selection and a reduction in their sensitivity to the delay interval itself. This deficit recovered with further training, which may be dependent upon relearning choice response procedures disrupted by the lesion, but might be reinstated by treatment with stimulant drugs. This article is part of a special issue entitled 'Behavioural, Anatomical, and Genetic Characterisation of Mouse and Rat Models of Huntington's Disease.'     

Effects of septal lesions on responding for delayed brain stimulation reinforcement

Rats with medial septal lesions were unable to learn in a situation where lateral hypothalamic stimulation reinforcement was delayed for 10 sec following each response. Rats with lateral septal lesions and rats with no lesions were able to learn under these conditions. All subjects in all groups exhibited high rates of responding for stimulation delivered concurrently with the response. The data suggest that a fiber system (possibly adrenergic) ascending from lateral hypothalamus through the medial septal area is involved in the mediation of brain stimulation reinforcement of complex tasks; and that some other neural mechanism which mediates the reinforcement of simple tasks must also exist.     

Neural mechanisms of negative reinforcement in children and adolescents with autism spectrum disorders

Background

Previous research has found accumulating evidence for atypical reward processing in autism spectrum disorders (ASD), particularly in the context of social rewards. Yet, this line of research has focused largely on positive social reinforcement, while little is known about the processing of negative reinforcement in individuals with ASD.

Methods

The present study examined neural responses to social negative reinforcement (a face displaying negative affect) and non-social negative reinforcement (monetary loss) in children with ASD relative to typically developing children, using functional magnetic resonance imaging (fMRI).

Results

We found that children with ASD demonstrated hypoactivation of the right caudate nucleus while anticipating non-social negative reinforcement and hypoactivation of a network of frontostriatal regions (including the nucleus accumbens, caudate nucleus, and putamen) while anticipating social negative reinforcement. In addition, activation of the right caudate nucleus during non-social negative reinforcement was associated with individual differences in social motivation.

Conclusions

These results suggest that atypical responding to negative reinforcement in children with ASD may contribute to social motivational deficits in this population.     

An effect of immediate reinforcement and delayed punishment, with possible implications for self-control

Behavior said to show self-control occurs virtually always as an alternative to behavior that produces conflicting consequences. One class of such consequences, immediate reinforcement and delayed punishment, is especially pervasive. Three experiments are described in which an effect of immediate reinforcement and delayed punishment is demonstrated. The results suggest that when immediate reinforcement and delayed punishment are imminent, the reinforcer alone controls the organism's behavior (in other words the organism behaves “impulsively”). The key to self-control, therefore, may be the acquisition of a large number of avoidance behaviors relevant to reinforcers that are correlated with delayed punishment. Human self-control may indeed involve such a process but undoubtedly involves others as well.     

Neural control of dopamine neurotransmission: implications for reinforcement learning

In the past few decades there has been remarkable convergence of machine learning with neurobiological understanding of reinforcement learning mechanisms, exemplified by temporal difference (TD) learning models. The anatomy of the basal ganglia provides a number of potential substrates for instantiation of the TD mechanism. In contrast to the traditional concept of direct and indirect pathway outputs from the striatum, we emphasize that projection neurons of the striatum are branched and individual striatofugal neurons innervate both globus pallidus externa and globus pallidus interna/substantia nigra (GPi/SNr). This suggests that the GPi/SNr has the necessary inputs to operate as the source of a TD signal. We also discuss the mechanism for the timing processes necessary for learning in the TD framework. The TD framework has been particularly successful in analysing electrophysiogical recordings from dopamine (DA) neurons during learning, in terms of reward prediction error. However, present understanding of the neural control of DA release is limited, and hence the neural mechanisms involved are incompletely understood. Inhibition is very conspicuously present among the inputs to the DA neurons, with inhibitory synapses accounting for the majority of synapses on DA neurons. Furthermore, synchronous firing of the DA neuron population requires disinhibition and excitation to occur together in a coordinated manner. We conclude that the inhibitory circuits impinging directly or indirectly on the DA neurons play a central role in the control of DA neuron activity and further investigation of these circuits may provide important insight into the biological mechanisms of reinforcement learning.     

Neural systems for error monitoring: recent findings and theoretical perspectives.

Complex behavior requires a flexible system that maintains task performance in the context of specific goals, evaluating behavioral progress, adjusting behavior as needed, and adapting to changing contingencies. Generically referred to as performance monitoring, a key component concerns the identification and correction of differences between an intended and an executed response (i.e., an error). Brain mapping experiments have now identified the temporal and spatial components of a putative error-processing system in the large-scale networks of the human brain. Most of this work has focused on the medial frontal cortex and an associated electrophysiological component known as the error-related negativity (or error negativity). Although the precise role, or roles, of this region still remain unknown, investigations of error processing have identified a cluster of modules in the medial frontal cortex involved in monitoring/maintaining ongoing behavior and motivating task sets. Other regions include bilateral anterior insula/inferior operculum and lateral prefrontal cortex. Recent work has begun to uncover how individual differences might affect the modules recruited for a task, in addition to the identification of associations between pathological states and aberrant error signals, leading to insights about possible mechanisms of neuropsychiatric illness.