Human subjects exploit a cognitive map for credit assignment

An influential reinforcement learning framework proposes that behavior is jointly governed by model-free (MF) and model-based (MB) controllers. The former learns the values of actions directly from past encounters, and the latter exploits a cognitive map of the task to calculate these prospectively. Considerable attention has been paid to how these systems interact during choice, but how and whether knowledge of a cognitive map contributes to the way MF and MB controllers assign credit (i.e., to how they revaluate actions and states following the receipt of an outcome) remains underexplored. Here, we examine such sophisticated credit assignment using a dual-outcome bandit task. We provide evidence that knowledge of a cognitive map influences credit assignment in both MF and MB systems, mediating subtly different aspects of apparent relevance. Specifically, we show MF credit assignment is enhanced for those rewards that are related to a choice, and this contrasted with choice-unrelated rewards that reinforced subsequent choices negatively. This modulation is only possible based on knowledge of task structure. On the other hand, MB credit assignment was boosted for outcomes that impacted on differences in values between offered bandits. We consider mechanistic accounts and the normative status of these findings. We suggest the findings extend the scope and sophistication of cognitive map-based credit assignment during reinforcement learning, with implications for understanding behavioral control.

An extensive body of psychological and neuroscientific literature on dual-system reinforcement learning (RL) indicates that behavior is governed by two distinct systems (1–17)—a rigid, retrospective model-free (MF) system (18, 19) and a flexible, prospective model-based (MB) system (18, 20). Unlike an MF system, which tends to repeat actions with a past history of success, an MB system relies on a cognitive map (CM) (21), that is, a model detailing the structure of a decision-making environment, including how states, actions, observations, and rewards are linked, to predict the impact of action choice on potential future rewards. Recent research highlights competitive and cooperative interactions between these systems, including speed accuracy trade-offs (22), reliability-based arbitration (1, 23), and a plan-to-habit strategy (24), with a focus on a prospective-planning role served by the MB system during choice. Recently, we demonstrated another influence of a CM (and thus, as we described it there, MB processes) in guiding credit assignment (CA) to MF action-values (i.e., affecting how MF values of actions and states are updated as reward-outcomes are received) (25). However, by design, this influence was limited to unraveling the resolution of state uncertainty for MF purposes, leaving broader aspects of the contribution of CM-based processes to CA unexplored.Here, we consider two potential complementary CM-based modulators of CA. Both concern the causal structure of the relationship between options and outcomes. One involves the “relatedness” of actual outcomes to an enacted choice, a retrospective effect of a CM on MF CA. The second involves the “importance” of potential outcomes during the deliberation process preceding a choice, a prospective effect of a CM on MB CA.“Relatedness” arises out of a complexity in assigning credit when information about streams of rewards is provided that depends only partly on the actions taken (unlike situations that involve simple lotteries, for instance, when an action is directly followed by the reward it occasions). An MF system, lacking structural causal knowledge, is disposed to assign credit naively to a choice based on the entire collection of ensuing outcomes, irrespective of whether these outcomes were caused by, or related to, an actual initiating action choice. By contrast, knowledge stored as a CM can guide MF CA to favor action-related outcomes.Take an example of a trader who deliberates purchasing one of two available mutual funds: X, which invests in companies A and B, or Y, which invests in companies A and C. Assume the trader opts for X and then later receives positive information about companies B and D. The trader might assign credit in an MF manner to her/his past action (“buy X”), updating the action’s cached value on the basis that positive consequences followed that choice. However, only one component of those positive consequences (that concerning company B) actually related to the choice of fund X. We propose that MF CA is modulated by a CM such that a change in the action’s value will be affected mostly by information about company B. More generally, relatedness depends on a causal attribution of rewards to actions (26).We consider a second modulator of CA, termed “importance,” as a form of attentional effect. When deliberating between several choice options, and taking into account their prospective outcomes, it is often the case that certain outcomes (which we dub “unimportant”) should not determine choice, as they are common to all choice options. In contrast, other (“important”) outcomes are distinctive to some choice options but not to others, and these should be the main determinants of choice. A CM will contain this type of information and direct attention to the latter alone. We consider the possibility that when the outcomes of the choice are observed, those that garnered more attention at choice are favored in learning.Consider our previous example where information about companies B and D triggers a CA process that leads to positive revaluation of these companies—a process useful for future MB financial decisions related to these companies. We propose this CA process can be biased by CM-based deliberations during choice. Notably, the values of companies B and C were “important” in the trader’s MB deliberation process (choosing a fund), as each is unique to one fund. The values of companies A and D, on the other hand, are less important, as these are either common to both choice options (A) or altogether absent (D). We hypothesize that representations of “important” components in a CM are activated more strongly during choice, leading to them being revalued more when information about choice outcomes are subsequently realized. Thus, ceteris paribus, the increase in the trader’s evaluation of company B will be higher than for company D, given the positive information. This evaluation is then exploited by MB planning processes for future choices.To test these hypotheses, we developed a variant of our previously described dual-outcome bandit task (25). Participants chose between pairs of bandits (i.e., lotteries) that led to different outcomes and received a stream of reward feedback pertaining to choice-related, choice-unrelated, important, and unimportant outcomes. Critically, there are two ways to value bandits in this task. An MF controller treats each bandit holistically, and, as described above, an MB controller predicts the values of the bandits from knowledge of the outcomes to which the bandits lead as provided by a putative CM. This distinction in the structure of evaluations can then be generalized to the apportioning of credit. We consider CA to a bandit to take the form of an MF credit assignment (MFCA; since the MF system makes decisions directly based on these values). Similarly, we consider CA to the outcomes associated with the bandits to be an MB credit assignment (MBCA). To put this another way, the main distinction between MFCA and MBCA in our task is that the former pertains to a revaluation of actions, while the latter pertains to a revaluation of latent causes for these actions (i.e., the ensuing outcomes).In support of our hypothesis that MFCA is guided by a CM, we found evidence that credit for choice-related and -unrelated outcomes is assigned to actions in a different manner. We show information about rewards actually related to chosen actions alone positively impact on the value of those actions. Information about rewards not related to chosen actions, on the other hand, have an opposite effect. Second, we found that MBCA was greater for choice outcomes that were “important” compared to “unimportant” during choice deliberations. We discuss mechanistic and normative accounts of these results.     

Open tibial fractures in major trauma centres: A national prospective cohort study of current practice

Aims

To assess current national practice in the management of severe open tibial fractures against national standards, using data collected by the Trauma and Audit Research Network.

Five cases of orbital extramedullary plasmacytoma: diagnosis and management of an aggressive malignancy

Purpose: Multiple myeloma is an insidious haematological malignancy characterised by monoclonal proliferation of plasma cells in the bone marrow. Extramedullary plasmacytoma is a rare manifestation of multiple myeloma and usually occurs in the upper respiratory tract. Orbital involvement is particularly uncommon, but may be associated with devastating visual impairment and poor clinical outcomes. Therefore, this article aims to highlight the need for multidisciplinary management of orbital extramedullary plasmacytoma.

Methods: This is a retrospective observational case series of five patients. All presented to the authors for management of orbital extramedullary plasmacytomas from 2004 to 2015 at Prince of Wales and Mater Hospitals in Sydney, Australia. Medical records were reviewed for pertinent information including demographics, disease features, management strategy, and clinical progress. The study met Medical Ethics Board standards and is in accordance with the Helsinki Agreements.

Results: This case series of five patients underscores the poor prognosis of orbital extramedullary plasmacytoma. Despite aggressive multidisciplinary management, four of these five patients succumbed to their illness during the study period. However, multidisciplinary management did manage to minimise symptoms and preserve quality of life.

Conclusions: On a case-by-case basis, patients may derive palliative benefit from orbital surgery in conjunction with radiotherapy and chemotherapy. Orbital surgeons are encouraged to work within a multidisciplinary framework of medical specialists, including haematologists and radiation oncologists, when determining the optimal management plan in cases of orbital extramedullary plasmacytoma.     

The mutational landscape of melanoma brain metastases presenting as the first visceral site of recurrence

Brain metastases are a major cause of melanoma-related mortality and morbidity. We undertook whole-exome sequencing of 50 tumours from patients undergoing surgical resection of brain metastases presenting as the first site of visceral disease spread and validated our findings in an independent dataset of 18 patients. Brain metastases had a similar driver mutational landscape to cutaneous melanomas in TCGA. However, KRAS was the most significantly enriched driver gene, with 4/50 (8%) of brain metastases harbouring non-synonymous mutations. Hotspot KRAS mutations were mutually exclusive from BRAF^V600, NRAS and HRAS mutations and were associated with a reduced overall survival from the resection of brain metastases (HR 10.01, p = 0.001). Mutations in KRAS were clonal and concordant with extracranial disease, suggesting that these mutations are likely present within the primary. Our analyses suggest that KRAS mutations could help identify patients with primary melanoma at higher risk of brain metastases who may benefit from more intensive, protracted surveillance.Subject terms: CNS cancer, Metastasis, Melanoma, Tumour biomarkers, Cancer     

Maternal personality traits,antenatal depressive symptoms and the postpartum mother–infant relationship: a prospective observational study

Social Psychiatry and Psychiatric Epidemiology - Maternal depression has been associated with bonding difficulties and lower maternal sensitivity in observed mother–infant interactions....     

The Association of a Mediterranean-Style Diet Pattern with Polycystic Ovary Syndrome Status in a Community Cohort Study

Polycystic ovary syndrome (PCOS) is a common condition in reproductive-aged women. While lifestyle management is first-line treatment in PCOS, the dietary intake of women with PCOS is unclear and there is no research assessing dietary patterns of women with and without PCOS. The aim of this study was to examine dietary patterns in a large cohort of women with and without PCOS. Data were from 7569 participants in the 1973–1978 cohort of the Australian Longitudinal Study on Women’s Health population assessed at 2009 (Survey 5) (n = 414 PCOS, n = 7155 non-PCOS). Dietary patterns were evaluated using factor analysis and multiple logistic regressions assessed their associations with PCOS status. Three dietary patterns were identified that explained 27% of the variance in food intake between women with and without PCOS: Non-core foods; Meats and take-away and Mediterranean-style. The Mediterranean-style dietary pattern was independently associated with PCOS status. On adjusted analysis for each 1 SD increase in the Mediterranean-style dietary pattern, there was a 26% greater likelihood that women had PCOS. This may indicate an improvement in the quality of dietary intake following a diagnosis of PCOS. Future research should examine the contribution of dietary patterns to the incidence and severity of PCOS and the potential for modification of dietary patterns in the lifestyle management of PCOS.