| Abstract: | Hippocampal cells are central to spatial and predictive representations, and experience replays by place cells are crucial for learning and memory. Nonetheless, how hippocampal replay patterns dynamically change during the learning process remains to be elucidated. Here, we designed a spatial task in which rats learned a new behavioral trajectory for reward. We found that as rats updated their behavioral strategies for a novel salient location, hippocampal cell ensembles increased theta-sequences and sharp wave ripple-associated synchronous spikes that preferentially replayed salient locations and reward-related contexts in reverse order. The directionality and contents of the replays progressively varied with learning, including an optimized path that had never been exploited by the animals, suggesting prioritized replays of significant experiences on a predictive map. Online feedback blockade of sharp wave ripples during a learning process inhibited stabilizing optimized behavior. These results implicate learning-associated experience replays that act to learn and reinforce specific behavioral strategies.To adapt to continuous changes in the external environment, animals need to encode new salient episodes and update their behavioral policy. To achieve flexible spatial navigation in particular, hippocampal place cells are thought to provide fundamental neural spatial coding and constitute a cognitive map (1). Recently, the view of the cognitive map has been proposed to extend to a predictive map theory in which place cells potentially encode expectations about an animal’s future states (2). In addition to these map representations, hippocampal place cells that encode animals’ past or future trajectories are sequentially activated during sharp wave ripples (SWRs) within a short time window (∼100 ms) in a phenomenon known as “place cell replays” (3–5). The time window of such SWR-associated sequential firing is potentially appropriate to induce plastic changes in the hippocampal circuit. Consistent with this idea, a growing amount of evidence demonstrates that the frequency of SWR-associated place cell replays (or reactivation) is prominently increased during learning of new experiences (online replays) (6) and during rest/sleep states after learning of new experiences (offline replays) (7). In addition, selective disruption of waking SWRs during spatial learning has been demonstrated to reduce subsequent task performance (8) and impair the stabilization and refinement of place cell maps (9). Taken together, these results suggest an essential role of SWR-associated replays of an animal’s experiences in novel spatial learning.Experience replays by place cells are also suggested to be a key neuronal basis for a reinforcement learning framework (10, 11). When agents encounter a prediction error, such as a change in reward, experience replays may be an efficient mechanism for the evaluation of their experienced action–outcome associations by providing a solution to the temporal credit assignment problem, which is helpful to update their behavioral policy to maximize future reward (12, 13). In line with this idea, empirical observations have demonstrated that the receipt of reward or novel experiences leads to increased rates of SWRs and coordinated reactivation of place cells (6, 14), and increased reward leads to increased reverse replays (4, 15).Despite accumulating evidence and theory, the field still lacks key insights into how the contents of place cell replays change to (re)assign the values of salient locations as animals develop new navigation strategies in response to environmental changes. Theoretical works demonstrate that incorporating replay algorithms with prioritized memory access for learned salient locations (16), rather than random access from all stored memory, into a reinforcement learning architecture—a strategy termed “prioritized experience replays” (17)—improves integrative learning in artificial agents. Whether living neuronal networks adopt the same computations to enhance learning capability remains an open question. To address this issue, we designed a spatial learning task requiring rats to learn new spatial navigation within a recording period within an hour, and analyzed how hippocampal replays of salient locations associated with learning progressively varied with the animal’s learning processes. |
