Human subjects exploit a cognitive map for credit assignment

An influential reinforcement learning framework proposes that behavior is jointly governed by model-free (MF) and model-based (MB) controllers. The former learns the values of actions directly from past encounters, and the latter exploits a cognitive map of the task to calculate these prospectively. Considerable attention has been paid to how these systems interact during choice, but how and whether knowledge of a cognitive map contributes to the way MF and MB controllers assign credit (i.e., to how they revaluate actions and states following the receipt of an outcome) remains underexplored. Here, we examine such sophisticated credit assignment using a dual-outcome bandit task. We provide evidence that knowledge of a cognitive map influences credit assignment in both MF and MB systems, mediating subtly different aspects of apparent relevance. Specifically, we show MF credit assignment is enhanced for those rewards that are related to a choice, and this contrasted with choice-unrelated rewards that reinforced subsequent choices negatively. This modulation is only possible based on knowledge of task structure. On the other hand, MB credit assignment was boosted for outcomes that impacted on differences in values between offered bandits. We consider mechanistic accounts and the normative status of these findings. We suggest the findings extend the scope and sophistication of cognitive map-based credit assignment during reinforcement learning, with implications for understanding behavioral control.

An extensive body of psychological and neuroscientific literature on dual-system reinforcement learning (RL) indicates that behavior is governed by two distinct systems (1–17)—a rigid, retrospective model-free (MF) system (18, 19) and a flexible, prospective model-based (MB) system (18, 20). Unlike an MF system, which tends to repeat actions with a past history of success, an MB system relies on a cognitive map (CM) (21), that is, a model detailing the structure of a decision-making environment, including how states, actions, observations, and rewards are linked, to predict the impact of action choice on potential future rewards. Recent research highlights competitive and cooperative interactions between these systems, including speed accuracy trade-offs (22), reliability-based arbitration (1, 23), and a plan-to-habit strategy (24), with a focus on a prospective-planning role served by the MB system during choice. Recently, we demonstrated another influence of a CM (and thus, as we described it there, MB processes) in guiding credit assignment (CA) to MF action-values (i.e., affecting how MF values of actions and states are updated as reward-outcomes are received) (25). However, by design, this influence was limited to unraveling the resolution of state uncertainty for MF purposes, leaving broader aspects of the contribution of CM-based processes to CA unexplored.Here, we consider two potential complementary CM-based modulators of CA. Both concern the causal structure of the relationship between options and outcomes. One involves the “relatedness” of actual outcomes to an enacted choice, a retrospective effect of a CM on MF CA. The second involves the “importance” of potential outcomes during the deliberation process preceding a choice, a prospective effect of a CM on MB CA.“Relatedness” arises out of a complexity in assigning credit when information about streams of rewards is provided that depends only partly on the actions taken (unlike situations that involve simple lotteries, for instance, when an action is directly followed by the reward it occasions). An MF system, lacking structural causal knowledge, is disposed to assign credit naively to a choice based on the entire collection of ensuing outcomes, irrespective of whether these outcomes were caused by, or related to, an actual initiating action choice. By contrast, knowledge stored as a CM can guide MF CA to favor action-related outcomes.Take an example of a trader who deliberates purchasing one of two available mutual funds: X, which invests in companies A and B, or Y, which invests in companies A and C. Assume the trader opts for X and then later receives positive information about companies B and D. The trader might assign credit in an MF manner to her/his past action (“buy X”), updating the action’s cached value on the basis that positive consequences followed that choice. However, only one component of those positive consequences (that concerning company B) actually related to the choice of fund X. We propose that MF CA is modulated by a CM such that a change in the action’s value will be affected mostly by information about company B. More generally, relatedness depends on a causal attribution of rewards to actions (26).We consider a second modulator of CA, termed “importance,” as a form of attentional effect. When deliberating between several choice options, and taking into account their prospective outcomes, it is often the case that certain outcomes (which we dub “unimportant”) should not determine choice, as they are common to all choice options. In contrast, other (“important”) outcomes are distinctive to some choice options but not to others, and these should be the main determinants of choice. A CM will contain this type of information and direct attention to the latter alone. We consider the possibility that when the outcomes of the choice are observed, those that garnered more attention at choice are favored in learning.Consider our previous example where information about companies B and D triggers a CA process that leads to positive revaluation of these companies—a process useful for future MB financial decisions related to these companies. We propose this CA process can be biased by CM-based deliberations during choice. Notably, the values of companies B and C were “important” in the trader’s MB deliberation process (choosing a fund), as each is unique to one fund. The values of companies A and D, on the other hand, are less important, as these are either common to both choice options (A) or altogether absent (D). We hypothesize that representations of “important” components in a CM are activated more strongly during choice, leading to them being revalued more when information about choice outcomes are subsequently realized. Thus, ceteris paribus, the increase in the trader’s evaluation of company B will be higher than for company D, given the positive information. This evaluation is then exploited by MB planning processes for future choices.To test these hypotheses, we developed a variant of our previously described dual-outcome bandit task (25). Participants chose between pairs of bandits (i.e., lotteries) that led to different outcomes and received a stream of reward feedback pertaining to choice-related, choice-unrelated, important, and unimportant outcomes. Critically, there are two ways to value bandits in this task. An MF controller treats each bandit holistically, and, as described above, an MB controller predicts the values of the bandits from knowledge of the outcomes to which the bandits lead as provided by a putative CM. This distinction in the structure of evaluations can then be generalized to the apportioning of credit. We consider CA to a bandit to take the form of an MF credit assignment (MFCA; since the MF system makes decisions directly based on these values). Similarly, we consider CA to the outcomes associated with the bandits to be an MB credit assignment (MBCA). To put this another way, the main distinction between MFCA and MBCA in our task is that the former pertains to a revaluation of actions, while the latter pertains to a revaluation of latent causes for these actions (i.e., the ensuing outcomes).In support of our hypothesis that MFCA is guided by a CM, we found evidence that credit for choice-related and -unrelated outcomes is assigned to actions in a different manner. We show information about rewards actually related to chosen actions alone positively impact on the value of those actions. Information about rewards not related to chosen actions, on the other hand, have an opposite effect. Second, we found that MBCA was greater for choice outcomes that were “important” compared to “unimportant” during choice deliberations. We discuss mechanistic and normative accounts of these results.     

Influence of calcitonin administration of serum calcium and inorganic phosphate level of the frog, Rana tigrina

The effect of salmon calcitonin (0.25 MRC mU/g body wt) was investigated on the serum calcium and inorganic phosphate levels of the frog. Rana tigrina. The hormone evokes hypocalcemia (on Day 1 and Day 3) which is followed by a significant hypercalcemia on Day 10. Thereafter, the level of calcium decreases again on Day 15. Calcitonin induces hypophosphatemia (on Day 3 and Day 5). Thereafter, hyperphosphatemia is recorded on Day 10. Normal serum phosphate value is achieved by Day 15. The results obtained in R. tigrina have been discussed in relation to the increased calcium deposits in the paravertebral lime sacs and to the possible enhanced secretion of the parathyroid glands.     

The correlation between ancestry and color in two cities of Northeast Brazil with contrasting ethnic compositions

The degree of admixture in Brazil between historically isolated populations is complex and geographically variable. Studies differ as to what the genetic and phenotypic consequences of this mixing have been. In Northeastern Brazil, we enrolled 522 residents of Salvador and 620 of Fortaleza whose distributions of self-declared color were comparable to those in the national census. Using the program Structure and principal components analysis there was a clear correlation between biogeographic ancestry and categories of skin color. This correlation with African ancestry was stronger in Salvador (r=0.585; P<0.001) than in Fortaleza (r=0.236; P<0.001). In Fortaleza, although self-declared blacks had a greater proportion of European ancestry, they had more African ancestry than the other categories. When the populations were analyzed without pseudoancestors, as in some studies, the relationship of ‘race'' to genetic ancestry tended to diffuse or disappear. The inclusion of different African populations also influenced ancestry estimates. The percentage of unlinked ancestry informative markers in linkage disequilibrium, a measure of population structure, was 3–5 times higher in both Brazilian populations than expected by chance. We propose that certain methods, ascertainment bias and population history of the specific populations surveyed can result in failure to demonstrate a correlation between skin color and genetic ancestry. Population structure in Brazil has important implications for genetic studies, but genetic ancestry is irrelevant for how individuals are treated in society, their health, their income or their inclusion. These track more closely with perceived skin color than genetic ancestry.     

Forest Structure and Community Composition of Cochin Mangroves,South-West Coast of India

This study presents the structural characteristics of Cochin mangroves in Kerala for proposing suitable management and rejuvenation measures of degrading mangrove habitats. The floristic diversity of mangroves revealed 14 species of true mangroves belonging to six families. Multivariate analysis of true mangroves belonging to selected mangrove forests of the study area based on density could be classified into four floristic groups, a water front or low–tide zonation, mid-tide preferring species, high tide and landward zonation. Shannon Weiner index of the three stations revealed that Site I was having higher value (H′ = 2.66) followed by Site II (H′ = 2.01) and Site III (H′ = 1.595). The density of the mangroves varied significantly with sites and species (Global R = 0.537, P < 0.001). The diameter at breast height (DBH) in the study area revealed that most of the species came under 1–10 cm DBH class. The overall structural data (including Importance Value Index, DBH and basal area) showed that Site III, the Mangalavanam forest was having more structural development and could be considered as matured forest whereas, Site I, Aroor is a maturing forest and Site II, Malippuram is the least matured forest from the study. The Importance Value Index and basal area of each species could be used for analysing the maturity of the forest and habitat preferences for restoration programmes of the degraded ecosystems.     

Prevalence of peripheral neuropathy among type 2 diabetes mellitus patients in a rural health centre in South India

International Journal of Diabetes in Developing Countries - There is a huge burden of diabetes-related complications, both microvascular and macrovascular, in India. With the rising prevalence of...     

Longer Time Interval Between Completion of Neoadjuvant Chemoradiation and Surgical Resection Does Not Improve Downstaging of Rectal Carcinoma

PURPOSE: An interval of six to eight weeks between completion of preoperative chemoradiation therapy and surgical resection of advanced rectal cancer has been described. Our purpose was to determine whether a longer time interval between completion of therapy and resection increases tumor downstaging and affects perioperative morbidity. METHODS: Forty patients with advanced adenocarcinoma of the rectum underwent preoperative chemoradiation on a prospective trial with irinotecan (50 mg/m2), 5-fluorouracil (225 mg/m2), and concomitant external-beam radiation (45-54 Gy) followed by complete surgical resection of the tumor with total mesorectal excision. The time interval between completion of chemoradiation and surgical resection ranged from 28 to 97 days. The patients were divided into two groups with 33 eligible patients: Group A (4-week to 8-week time interval; 28-56 days) and Group B (10-week to 14-week interval; 67-97 days). Tumor downstaging was compared between these two groups. The number of patients downstaged by at least one T stage, those downstaged by at least one N stage, those with pathologic complete responses, and those with only residual microscopic tumor foci were compared. Postoperative length of stay, estimated blood loss, perioperative morbidity, and sphincter-sparing procedures were also compared. Chi-squared tests and Student's t-test were calculated. RESULTS: Group A had 19 patients, and Group B had 14 patients. Patient demographics were comparable. Mean age was 52 years, and 70 percent of patients were male. There were no deaths. There were no statistical differences in perioperative morbidity, with three anastomotic leaks in Group A. Tumors were downstaged in 58 percent of patients in Group A and 43 percent of those in Group B (P = 0.61). Nodal downstaging occurred in 78 percent of Group A and 67 percent of Group B (P = 0.9). The pathologic complete response rate was 21 percent in Group A and 14 percent in Group B (P = 0.97), and a residual microfocus of tumor was found in 33 percent of patients in Group A and 42 percent of those in Group B (P = 0.90). These differences were not statistically significant. CONCLUSIONS: Perioperative morbidity is not affected by longer intervals. A longer interval between completion of neoadjuvant chemoradiation and surgical resection may not increase the tumor response rate of advanced rectal cancer in this cohort.