Human subjects exploit a cognitive map for credit assignment

An influential reinforcement learning framework proposes that behavior is jointly governed by model-free (MF) and model-based (MB) controllers. The former learns the values of actions directly from past encounters, and the latter exploits a cognitive map of the task to calculate these prospectively. Considerable attention has been paid to how these systems interact during choice, but how and whether knowledge of a cognitive map contributes to the way MF and MB controllers assign credit (i.e., to how they revaluate actions and states following the receipt of an outcome) remains underexplored. Here, we examine such sophisticated credit assignment using a dual-outcome bandit task. We provide evidence that knowledge of a cognitive map influences credit assignment in both MF and MB systems, mediating subtly different aspects of apparent relevance. Specifically, we show MF credit assignment is enhanced for those rewards that are related to a choice, and this contrasted with choice-unrelated rewards that reinforced subsequent choices negatively. This modulation is only possible based on knowledge of task structure. On the other hand, MB credit assignment was boosted for outcomes that impacted on differences in values between offered bandits. We consider mechanistic accounts and the normative status of these findings. We suggest the findings extend the scope and sophistication of cognitive map-based credit assignment during reinforcement learning, with implications for understanding behavioral control.

An extensive body of psychological and neuroscientific literature on dual-system reinforcement learning (RL) indicates that behavior is governed by two distinct systems (1–17)—a rigid, retrospective model-free (MF) system (18, 19) and a flexible, prospective model-based (MB) system (18, 20). Unlike an MF system, which tends to repeat actions with a past history of success, an MB system relies on a cognitive map (CM) (21), that is, a model detailing the structure of a decision-making environment, including how states, actions, observations, and rewards are linked, to predict the impact of action choice on potential future rewards. Recent research highlights competitive and cooperative interactions between these systems, including speed accuracy trade-offs (22), reliability-based arbitration (1, 23), and a plan-to-habit strategy (24), with a focus on a prospective-planning role served by the MB system during choice. Recently, we demonstrated another influence of a CM (and thus, as we described it there, MB processes) in guiding credit assignment (CA) to MF action-values (i.e., affecting how MF values of actions and states are updated as reward-outcomes are received) (25). However, by design, this influence was limited to unraveling the resolution of state uncertainty for MF purposes, leaving broader aspects of the contribution of CM-based processes to CA unexplored.Here, we consider two potential complementary CM-based modulators of CA. Both concern the causal structure of the relationship between options and outcomes. One involves the “relatedness” of actual outcomes to an enacted choice, a retrospective effect of a CM on MF CA. The second involves the “importance” of potential outcomes during the deliberation process preceding a choice, a prospective effect of a CM on MB CA.“Relatedness” arises out of a complexity in assigning credit when information about streams of rewards is provided that depends only partly on the actions taken (unlike situations that involve simple lotteries, for instance, when an action is directly followed by the reward it occasions). An MF system, lacking structural causal knowledge, is disposed to assign credit naively to a choice based on the entire collection of ensuing outcomes, irrespective of whether these outcomes were caused by, or related to, an actual initiating action choice. By contrast, knowledge stored as a CM can guide MF CA to favor action-related outcomes.Take an example of a trader who deliberates purchasing one of two available mutual funds: X, which invests in companies A and B, or Y, which invests in companies A and C. Assume the trader opts for X and then later receives positive information about companies B and D. The trader might assign credit in an MF manner to her/his past action (“buy X”), updating the action’s cached value on the basis that positive consequences followed that choice. However, only one component of those positive consequences (that concerning company B) actually related to the choice of fund X. We propose that MF CA is modulated by a CM such that a change in the action’s value will be affected mostly by information about company B. More generally, relatedness depends on a causal attribution of rewards to actions (26).We consider a second modulator of CA, termed “importance,” as a form of attentional effect. When deliberating between several choice options, and taking into account their prospective outcomes, it is often the case that certain outcomes (which we dub “unimportant”) should not determine choice, as they are common to all choice options. In contrast, other (“important”) outcomes are distinctive to some choice options but not to others, and these should be the main determinants of choice. A CM will contain this type of information and direct attention to the latter alone. We consider the possibility that when the outcomes of the choice are observed, those that garnered more attention at choice are favored in learning.Consider our previous example where information about companies B and D triggers a CA process that leads to positive revaluation of these companies—a process useful for future MB financial decisions related to these companies. We propose this CA process can be biased by CM-based deliberations during choice. Notably, the values of companies B and C were “important” in the trader’s MB deliberation process (choosing a fund), as each is unique to one fund. The values of companies A and D, on the other hand, are less important, as these are either common to both choice options (A) or altogether absent (D). We hypothesize that representations of “important” components in a CM are activated more strongly during choice, leading to them being revalued more when information about choice outcomes are subsequently realized. Thus, ceteris paribus, the increase in the trader’s evaluation of company B will be higher than for company D, given the positive information. This evaluation is then exploited by MB planning processes for future choices.To test these hypotheses, we developed a variant of our previously described dual-outcome bandit task (25). Participants chose between pairs of bandits (i.e., lotteries) that led to different outcomes and received a stream of reward feedback pertaining to choice-related, choice-unrelated, important, and unimportant outcomes. Critically, there are two ways to value bandits in this task. An MF controller treats each bandit holistically, and, as described above, an MB controller predicts the values of the bandits from knowledge of the outcomes to which the bandits lead as provided by a putative CM. This distinction in the structure of evaluations can then be generalized to the apportioning of credit. We consider CA to a bandit to take the form of an MF credit assignment (MFCA; since the MF system makes decisions directly based on these values). Similarly, we consider CA to the outcomes associated with the bandits to be an MB credit assignment (MBCA). To put this another way, the main distinction between MFCA and MBCA in our task is that the former pertains to a revaluation of actions, while the latter pertains to a revaluation of latent causes for these actions (i.e., the ensuing outcomes).In support of our hypothesis that MFCA is guided by a CM, we found evidence that credit for choice-related and -unrelated outcomes is assigned to actions in a different manner. We show information about rewards actually related to chosen actions alone positively impact on the value of those actions. Information about rewards not related to chosen actions, on the other hand, have an opposite effect. Second, we found that MBCA was greater for choice outcomes that were “important” compared to “unimportant” during choice deliberations. We discuss mechanistic and normative accounts of these results.     

Five cases of orbital extramedullary plasmacytoma: diagnosis and management of an aggressive malignancy

Purpose: Multiple myeloma is an insidious haematological malignancy characterised by monoclonal proliferation of plasma cells in the bone marrow. Extramedullary plasmacytoma is a rare manifestation of multiple myeloma and usually occurs in the upper respiratory tract. Orbital involvement is particularly uncommon, but may be associated with devastating visual impairment and poor clinical outcomes. Therefore, this article aims to highlight the need for multidisciplinary management of orbital extramedullary plasmacytoma.

Methods: This is a retrospective observational case series of five patients. All presented to the authors for management of orbital extramedullary plasmacytomas from 2004 to 2015 at Prince of Wales and Mater Hospitals in Sydney, Australia. Medical records were reviewed for pertinent information including demographics, disease features, management strategy, and clinical progress. The study met Medical Ethics Board standards and is in accordance with the Helsinki Agreements.

Results: This case series of five patients underscores the poor prognosis of orbital extramedullary plasmacytoma. Despite aggressive multidisciplinary management, four of these five patients succumbed to their illness during the study period. However, multidisciplinary management did manage to minimise symptoms and preserve quality of life.

Conclusions: On a case-by-case basis, patients may derive palliative benefit from orbital surgery in conjunction with radiotherapy and chemotherapy. Orbital surgeons are encouraged to work within a multidisciplinary framework of medical specialists, including haematologists and radiation oncologists, when determining the optimal management plan in cases of orbital extramedullary plasmacytoma.     

Communication,collaboration and contagion: “Virtualisation” of anatomy during COVID‐19

COVID‐19 has generated a global need for technologies that enable communication, collaboration, education and scientific discourse whilst maintaining physical distance. University closures due to COVID‐19 and physical distancing measures disrupt academic activities that previously occurred face‐to‐face. Restrictions placed on universities due to COVID‐19 have precluded most conventional forms of education, assessment, research and scientific discourse. Anatomists now require valid, robust and easy‐to‐use communication tools to facilitate remote teaching, learning and research. Recent advances in communication, video conferencing and digital technologies may facilitate continuity of teaching and research activities. Examples include highly‐interactive video conferencing technology, collaborative tools, social media and networking platforms. In this narrative review, we examine the utility of these technologies in supporting effective communication and professional activities of anatomists during COVID‐19 and after.