Model-free decision making is prioritized when learning to avoid harming others

Moral behavior requires learning how our actions help or harm others. Theoretical accounts of learning propose a key division between “model-free” algorithms that cache outcome values in actions and “model-based” algorithms that map actions to outcomes. Here, we tested the engagement of these mechanisms and their neural basis as participants learned to avoid painful electric shocks for themselves and a stranger. We found that model-free decision making was prioritized when learning to avoid harming others compared to oneself. Model-free prediction errors for others relative to self were tracked in the thalamus/caudate. At the time of choice, neural activity consistent with model-free moral learning was observed in subgenual anterior cingulate cortex (sgACC), and switching after harming others was associated with stronger connectivity between sgACC and dorsolateral prefrontal cortex. Finally, model-free moral learning varied with individual differences in moral judgment. Our findings suggest moral learning favors efficiency over flexibility and is underpinned by specific neural mechanisms.

A central component of human morality is a prohibition against harming others (1, 2). People readily avoid actions that might harm another person (3–7), and this basic harm aversion is so strong that many people even find it distressing to perform pretend harmful actions, such as shooting someone with a fake gun (8). Harm aversion is disrupted in clinical disorders such as psychopathy that have a strong developmental component (9), and although harm aversion is robust in healthy adults, anyone who has watched young children fighting over a coveted toy knows that such an aversion is not present from birth. Indeed, a large literature documents the emergence of moral conduct over the course of development (7, 10, 11). Cross-cultural differences in morality suggest moral behavior is fine-tuned to local environmental demands (12), and laboratory experiments demonstrate how individuals can quickly adapt moral behavior to changing norms (13, 14). All this evidence highlights a critical role for learning in the development of harm aversion and moral behavior more broadly (6). Once having learned as children that harming others is morally wrong, adults still need to learn which actions to take to avoid harm in novel contexts.Recent work in computational neuroscience has advanced our knowledge of how organisms learn the value of actions and outcomes via reward and punishment (15, 16). An important theoretical distinction has been made between “model-based” and “model-free” learning systems (17, 18). Model-based learning is often described as deliberative learning, whereas model-free learning is thought to be habitual. The model-based system builds a “world model” of the environment and selects actions by prospectively searching the model for the best course of action (19, 20). In contrast, the computationally efficient model-free system assigns values to actions simply through trial and error. The distinction between these systems can be illustrated by giving the example of how we navigate home from work. The model-based system could easily replan if a particular route home was unexpectedly blocked, whereas a purely model-free learner can only plan a route home by directly experiencing each of the different routes (21). These two systems are also somewhat neurally dissociable, with model-based learning preferentially engaging lateral prefrontal cortex (LPFC), posterior parietal cortex, and caudate (20, 22, 23) and model-free learning preferentially engaging putamen (24, 25), although both systems update their representations via prediction errors encoded in overlapping regions of ventral striatum (20). Model-based and model-free systems often make similar recommendations about which actions are more valuable, but when they conflict an arbitration process allocates control between them (12, 13, 23, 26, 27). However, despite extensive theorizing that the model-based/model-free distinction may help to characterize puzzling features of moral learning and decision making (3, 28–30), it remains unknown whether the moral consequences of actions affect the balance between model-based and model-free control, and whether common or distinct neural processes are engaged when learning to avoid harmful outcomes to self and others.Past work on the neural basis of moral decision making provides support for competing hypotheses. On the one hand, the sophistication of human morality seems to demand the kinds of complex representations afforded by model-based learning, suggesting learning to avoid harming others may preferentially engage the model-based system. Supporting this view, people are easily able to learn to avoid harmful actions without directly experiencing their outcomes, in line with a model-based learning strategy when avoiding harm to others (3, 28, 31). Moreover, moral decision making in healthy adults consistently engages brain regions most strongly associated with the model-based system, including LPFC, caudate, and temporoparietal junction (TPJ) (24, 26, 32). Deciding to follow moral norms like fairness and honesty, and enforcing those norms on others via costly punishment, engages LPFC (33–38), and disrupting LPFC function reduces moral norm compliance and enforcement (39, 40). During decisions to avoid harming others, LPFC encodes the blameworthiness of harmful choices and modulates action values in caudate and thalamus (4), two subcortical areas shown to play a critical role in associative learning and pain processing as well as moral decision making (41–46).On the other hand, one principal function of model-free learning is to cache value in actions that are reliably adaptive, sacrificing flexibility for efficiency. Given that harming others is typically prohibited, actions that harm others may represent a special class of actions that are prioritized for model-free learning, similar to how certain classes of stimuli, like snakes and spiders, are “prepared” for aversive classical conditioning (47). In other words, since avoiding harm to others is hugely important for social life, learning processes that fast-track harm-avoidant action selection to a habitual, automatic process may be socially adaptive. Supporting this view, recent work suggests that morality constrains mental representations of what actions are considered possible; harmful actions are removed from choice sets as a default (48), and choices that harm others are slower than helpful choices, suggesting an automatic tendency to avoid harm (5, 49–51). Furthermore, recent studies of model-free learning to gain rewards for oneself and others have highlighted a distinct encoding of prediction errors concerning others’ outcomes in the subgenual anterior cingulate cortex (sgACC) (52, 53), a region that has been implicated in social and moral decision making more broadly (53–57). Model-free processes that distinguish learning about how one’s actions affect others could provide a neural mechanism for prioritizing model-free learning in moral contexts.To test these competing hypotheses, we used computational modeling and functional MRI (fMRI) to probe the relative balance between model-based vs. model-free processes, and their neural bases, when people learn to avoid moderately painful electric shocks for themselves and a stranger. Forty-one participants attended a 3.5-h experimental session. After undergoing an extensive pain thresholding procedure (Methods), they completed a hybrid version of two paradigms previously proposed to reliably dissociate model-free vs. model-based learning (Fig. 1) (20, 23, 32). We optimized the task in a way that allowed us to address the specific hypotheses examined in the present study (see SI Appendix, Supplementary Text for details) and included as many as 272 trials per participant to accurately sample decisions for both self and other. Our final analysis included 36 participants who made a total of 9,792 choices.Open in a separate windowFig. 1.Model-free and model-based aversive learning task. Participants completed a two-stage decision-making task to assess the tendency to engage in model-free and model-based learning. The task was a hybrid of two tasks previously shown to assess model-free and model-based learning processes (20, 27). We used this task to probe learning to avoid aversive (shock) outcomes for either oneself or another person (the “receiver,” referred to as “other” hereafter). At the beginning of each block, an instruction cue signaled the recipient of the outcome (self or other). At the first stage, two images were displayed that probabilistically led to one of two states (common [∼70% of the time] or uncommon [rare] transition [∼30% of the time]), depicted by different colors surrounding the boxes. In this example, to “blue zone” or “yellow zone” for the other participant and “turquoise zone” or “purple zone” for self. Participants then made a second choice between two pictures in the colored zone which was followed by an outcome of shock or no shock. The probability with which the boxes at the second stage delivered a shock or no-shock outcome drifted throughout the experiment (bounded between 0 and 1 with a drift rate of 0.2) and participants were instructed to keep learning throughout. Ten percent of the total electric shocks accumulated in the “self” condition were delivered to the participant themselves at the end of the experiment, while 10% of the electric shocks accumulated in the “other” condition were delivered to the partner participant.