Investigating the functional interaction between semantic and episodic memory: convergent behavioral and electrophysiological evidence for the role of familiarity

Throughout our lives we acquire general knowledge about the world (semantic memory) while also retaining memories of specific events (episodic memory). Although these two forms of memory have been dissociated on the basis of neuropsychological data, it is clear that they typically function together during normal cognition. The goal of the present study was to investigate this interaction. One influence of semantic memory on episodic retrieval is 'Levels Of Processing'; recognition is enhanced when stimuli are processed in a semantically meaningful way. Studies examining this semantic processing advantage have largely concluded that semantic memory augments episodic retrieval primarily by enhancing recollection. The present study provides strong evidence for an alternative relationship between semantic and episodic memory. We employed a manipulation of the semantic coherence of to-be-remembered information (semantically related vs. unrelated word pairs) during an associative recognition memory test. Results revealed that associative recognition is significantly enhanced for semantically coherent material, and behavioral estimates (using the process dissociation procedure) demonstrated concomitant changes in the contribution of familiarity to retrieval. Neuroimaging data (event-related potentials recorded at test) also revealed a significant increase in familiarity based retrieval. The electrophysiological correlate of familiarity (the mid-frontal ERP old/new effect) was larger for semantically related compared to unrelated word pairs, but no difference was present in the electrophysiological correlate of recollection (the left parietal old/new effect). We conclude that semantic memory and episodic memory do indeed interact in normal functioning, and not only by modulating recollection, but also by enhancing familiarity.     

Neural correlates of successful encoding of semantically and phonologically mediated inter-item associations

This experiment investigated whether the neural correlates of inter-item associative encoding vary according to study task. At study, pairs of unrelated words were subjected to either semantic or phonological relational judgments. Test items comprised studied word pairs (intact), pairs comprised of words belonging to different study pairs (rearranged), and novel pairs. The test requirement was to discriminate between these different classes of test item. fMRI was employed to contrast the neural activity elicited by studied pairs that were correctly endorsed as intact on the later associative recognition test, as opposed to pairs for which associative information was unavailable. In contrast to prior findings for the encoding of single items, there was no evidence that the loci of subsequent associative memory effects varied according to study task. Instead, in both tasks, pairs that were later correctly endorsed as intact elicited enhanced activity in mid- and ventral regions of the left ventrolateral prefrontal cortex (VLPFC). These findings were accompanied by extensive task-invariant reversed subsequent memory effects in medial and lateral parietal and frontal cortices. The findings suggest that the left VLPFC may play a domain-general role in the encoding of item-item associations, and in addition highlight the importance of elucidating the functional significance of reversed subsequent memory effects.     

Examining ERP correlates of recognition memory: evidence of accurate source recognition without recollection

Recollection is typically associated with high recognition confidence and accurate source memory. However, subjects sometimes make accurate source memory judgments even for items that are not confidently recognized, and it is not known whether these responses are based on recollection or some other memory process. In the current study, we measured event related potentials (ERPs) while subjects made item and source memory confidence judgments in order to determine whether recollection supported accurate source recognition responses for items that were not confidently recognized. In line with previous studies, we found that recognition memory was associated with two ERP effects: an early on-setting FN400 effect, and a later parietal old-new effect [late positive component (LPC)], which have been associated with familiarity and recollection, respectively. The FN400 increased gradually with item recognition confidence, whereas the LPC was only observed for highly confident recognition responses. The LPC was also related to source accuracy, but only for items that had received a high confidence item recognition response; accurate source judgments to items that were less confidently recognized did not exhibit the typical ERP correlate of recollection or familiarity, but rather showed a late, broadly distributed negative ERP difference. The results indicate that accurate source judgments of episodic context can occur even when recollection fails.     

Dissociating recollection from familiarity: electrophysiological evidence that familiarity for faces is associated with a posterior old/new effect

In recognition memory research, a tension exists between dual-process and single-process models of episodic retrieval. Dual-process models propose that 'familiarity' assessment and the 'recollection' of contextual information are independent processes, while single-process models claim that one common process supports retrieval. Event-related potentials (ERPs) have been used to show dissociations between the mid frontal and the left parietal ERP old/new effects, which have been associated with familiarity and recollection, respectively. While much ERP evidence favours dual-process theory, Yovel and Paller [Yovel, G., Paller, K.A., 2004. The neural basis of the butcher-on-the-bus phenomenon: when a face seems familiar but is not remembered. NeuroImage 21, 789-800] used faces as retrieval cues to demonstrate that posterior old/new effects index both familiarity and recollection, a finding consistent with single-process models. Here we present evidence supporting Yovel and Paller's claim that a posterior old/new effect indexes familiarity for faces, along with a novel finding that recollection is associated with an anterior old/new effect. Importantly, and in contrast to Yovel and Paller, the old/new effects associated with familiarity and recollection were topographically dissociable, consistent with a dual-process view of recognition memory. The neural correlates of familiarity and recollection identified here for faces appear to be different from those typically observed, suggesting that the ERP old/new effects associated with episodic recognition are not the same under all circumstances.     

Changes in brain electrical activity during extended continuous word recognition

Twenty healthy subjects (10 men, 10 women) participated in an EEG study with an extended continuous recognition memory task, in which each of 30 words was randomly shown 10 times and subjects were required to make old vs. new decisions. Both event-related brain potentials (ERPs) and induced band power (IBP) were investigated. We hypothesized that repeated presentations affect recollection rather than familiarity. For the 300- to 500-ms time window, an 'old/new' ERP effect was found for the first vs. second word presentations. The correct recognition of an 'old' word was associated with a more positive waveform than the correct identification of a new word. The old/new effect was most pronounced at and around the midline parietal electrode position. For the 500- to 800-ms time window, a linear repetition effect was found for multiple word repetitions. Correct recognition after an increasing number of repetitions was associated with increasing positivity. The multiple repetitions effect was most pronounced at the midline central (Cz) and fronto-central (FCz) electrode positions and reflects a graded recollection process: the stronger the memory trace grows, the more positive the ERP in the 500- to 800-ms time window. The ERP results support a dual-processing model, with familiarity being discernable from a more graded recollection state that depends on memory strengths. For IBP, we found 'old/new' effects for the lower-2 alpha, theta, and delta bands, with higher bandpower during 'old' words. The lower-2 alpha 'old/new' effect most probably reflects attentional processes, whereas the theta and delta effects reflect encoding and retrieval processes. Upon repeated word presentations, the magnitude of induced delta power in the 375- to 750-ms time window diminished linearly. Correlation analysis suggests that decreased delta power is moderately associated with faster decision speed and higher accuracy.     

Content dependence of the electrophysiological correlates of recollection

The neural correlates of episodic retrieval ('recollection') have been shown to differ according to the content of retrieved episodes. It has been hypothesized that these content-dependent differences reflect the 'reinstatement' of encoding-related processes or representations at the time of recollection. It remains unclear, however, whether these effects directly reflect the recollection of differential episodic content, as would be predicted by the reinstatement hypothesis, or whether they are instead associated with processes that are contingent on successful recollection. To address this issue, the present study employed event-related potentials (ERPs), permitting the investigation of the temporal dynamics of content-dependent neural effects during retrieval, and in particular, their onset with respect to well-established ERP correlates of recollection, such as the left parietal old-new effect. Subjects studied a series of words that were each presented in the context of one of two encoding tasks. One task required the covert generation of a sentence incorporating each word, whereas the other required imagining the object corresponding to each word within a superimposed scenic picture. Memory for the words was subsequently tested with the 'remember/know' procedure. ERPs elicited by recollected words differed according to the prior encoding history of the word, beginning at approximately 300 ms following word onset. These content-dependent ERP differences were maximal over the anterior scalp and, importantly, onset as early as the left parietal old-new effect. The findings demonstrate that content-dependent neural activity during retrieval can occur in a timeframe that is compatible with a direct role in the recollection and representation of episodic information.     

The FN400 is functionally distinct from the N400

The FN400 refers to the early midfrontally-distributed difference between ERPs elicited by old and new items, which operates in a way consistent with a neural marker of familiarity-based recognition. Double dissociations between the FN400 and a later ERP index of recollection provide some of the most compelling evidence in support of dual-process models to date. It has recently been claimed, however, that there is no evidence that the FN400 is functionally distinct from the N400 index of implicit semantic priming (Voss, J., and Federmeier, K., FN400 potentials are functionally identical to N400 potentials and reflect semantic processing during recognition testing, Psychophysiology, 48, 532-546, 2011), challenging inferences made on the basis of this effect. We argue that the design employed to make this claim is flawed because it comprised a semantic priming manipulation embedded within a continuous recognition test which enabled recognition contrasts to be confounded by semantic processes in a number of ways. Here, ERPs were recorded from a design which avoided these confounds by employing a semantic priming paradigm which also served as the encoding phase for a surprise subsequent recognition test phase. An N400 effect elicited in the semantic priming task demonstrated the established centro-parietal maximum, whereas the difference between correctly responded to old and new ERPs in the recognition test was maximal over frontal sites in the same time window. When direct comparisons of the electrophysiological correlates of semantic priming and episodic recognition are recorded in a paradigm in which the two are not confounded, the FN400 reflects a qualitatively distinct effect from the N400.     

The neural basis of the butcher-on-the-bus phenomenon: when a face seems familiar but is not remembered

A common distinction in contemporary research on episodic memory is between familiarity, an unsubstantiated impression that an event was experienced previously, and recollection, remembering some information plus the spatiotemporal context of the episode in which it was acquired. The epitome of pure familiarity--the butcher-on-the-bus phenomenon--occurs when one believes that a person is familiar (often upon seeing their face in an atypical context) while failing to recall any information about that person whatsoever. Prior research on familiarity and recollection has relied on verbal material. Whereas word meanings and pronunciations are well learned in advance, here we produced pure familiarity and recollection using photographs of faces never seen before the experiment. When participants recognized a face, recollection was inferred if they also remembered either the occupation associated with that face earlier in the experiment or any other episodic detail. Pure familiarity was inferred when recognition occurred in the absence of any such contextual retrieval. Analyses of brain potentials recorded during initial encoding showed that right-sided neural activity predicted subsequent face familiarity, whereas bilateral potentials predicted subsequent face recollection. Results during memory testing were inconsistent with the popular idea that familiarity is generically indexed by reduced frontal N400-like potentials. Instead, both memory experiences were associated with bilateral, parietal-maximum brain potentials, although with smaller amplitudes and for a shorter duration for familiarity. These similarities between electrophysiological correlates of pure familiarity and recollection suggest that familiarity with faces may arise by virtue of a subset of the neural processing responsible for recollection.     

Developmental changes in the neural correlates of semantic processing

Functional magnetic resonance imaging (fMRI) was used to explore the neural correlates of semantic judgments in the auditory modality in a group of 9- to 15-year-old children. Subjects were required to indicate if word pairs were related in meaning. Consistent with previous findings in adults, children showed activation in bilateral superior temporal gyri (BA 22) for recognizing spoken words as well as activations in bilateral inferior frontal gyri (BAs 47, 45) and left middle temporal gyrus (BA 21) for semantic processing. The neural substrates of semantic association and age differences were also investigated. Words with strong semantic association elicited significantly greater activation in the left inferior parietal lobule (BA 40), whereas words with weak semantic association elicited activation in left inferior frontal gyrus (BAs 47/45). Correlations with age were observed in the left middle temporal gyrus (BA 21) and the right inferior frontal gyrus (BA 47). The pattern of results for semantic association implies that the left inferior parietal lobule effectively integrates highly related semantic features and the left inferior frontal gyrus becomes more active for words that require a greater search for semantic associations. The developmental results suggest that older children recruit the right inferior frontal gyrus as they conduct a broader semantic search and the left middle temporal gyrus to provide more efficient access to semantic representations.     

Coherent theta-band EEG activity predicts item-context binding during encoding

Episodic memories consist of semantic information coupled with a rich array of contextual detail. Here, we investigate the neural processes by which information about the sensory context of a learning event is "bound" to the semantic representation of the to-be-encoded item. We present evidence that item-context binding during encoding is mediated by frontoposterior electroencephalographic (EEG) phase locking within and between hemispheres in the theta (4-8 Hz) band. During a task in which subjects encoded words in different font colors, later memory for the word was associated with sustained frontal theta activity and frontoposterior theta-band coherence, primarily within the left hemisphere. When the word-color association was later successfully retrieved, however, neurons synchronized their theta-band responses bilaterally in a more sustained fashion, particularly during the latter part of the stimulus epoch (>800 ms). Our results confirm the importance of functional coupling between frontal and posterior regions for successful encoding. One interpretation of these data is hemispheric contributions to item and context encoding may be asymmetric, with left hemisphere coherence facilitating semantic processing of an item and right hemisphere coherence facilitating processing of sensory context. Theta-band coherence may be an important mechanism by which brain networks exchange information during learning.     

Effects of semantic relatedness on age-related associative memory deficits: the role of theta oscillations

Growing evidence suggests that age-related deficits in associative memory are alleviated when the to-be-associated items are semantically related. Here we investigate whether this beneficial effect of semantic relatedness is paralleled by spatio-temporal changes in cortical EEG dynamics during incidental encoding. Young and older adults were presented with faces at a particular spatial location preceded by a biographical cue that was either semantically related or unrelated. As expected, automatic encoding of face-location associations benefited from semantic relatedness in the two groups of age. This effect correlated with increased power of theta oscillations over medial and anterior lateral regions of the prefrontal cortex (PFC) and lateral regions of the posterior parietal cortex (PPC) in both groups. But better-performing elders also showed increased brain-behavior correlation in the theta band over the right inferior frontal gyrus (IFG) as compared to young adults. Semantic relatedness was, however, insufficient to fully eliminate age-related differences in associative memory. In line with this finding, poorer-performing elders relative to young adults showed significant reductions of theta power in the left IFG that were further predictive of behavioral impairment in the recognition task. All together, these results suggest that older adults benefit less than young adults from executive processes during encoding mainly due to neural inefficiency over regions of the left ventrolateral prefrontal cortex (VLPFC). But this associative deficit may be partially compensated for by engaging preexistent semantic knowledge, which likely leads to an efficient recruitment of attentional and integration processes supported by the left PPC and left anterior PFC respectively, together with neural compensatory mechanisms governed by the right VLPFC.     

On why the elderly have normal semantic retrieval but deficient episodic encoding: a study of left inferior frontal ERP activity

Age-related left inferior prefrontal cortex (LIPFC) blood flow reductions during semantic retrieval are associated with reduced subsequent episodic recognition memory performance but are inconsistent with age-invariant semantic retrieval performance. Therefore, we compared brain activity in young and elderly persons during low- and high-selection versions of a semantic task using ERPs recorded at 62 scalp locations. In an early interval (400-800 ms), both age groups showed more negativity over left inferior frontal scalp while performing the high- compared to the low-selection task. This early semantic selection-related negativity was associated with age-equivalent accuracy and reaction time on the semantic tasks. Further, in the early time interval, the ERPs of the young were more negative than those of the elderly. In addition, only the young showed a late selection-related negativity (1,200--1,400 ms). These age-related differences in left frontal ERP activity were associated with significant decrements in subsequent recognition for the elderly. Moreover, additional analyses revealed that larger amounts of negativity over left inferior frontal scalp locations during semantic retrieval were correlated with enhanced subsequent episodic recognition in both groups. In sum, the data suggest that (1) semantic retrieval and episodic encoding processes overlap temporally in similar brain regions, most likely including the LIPFC, and (2) the failure to recruit these regions in the service of episodic encoding underlies the age-related deficit in episodic memory.     

Left temporal and temporoparietal brain activity depends on depth of word encoding: a magnetoencephalographic study in healthy young subjects

Using a 143-channel whole-head magnetoencephalograph (MEG) we recorded the temporal changes of brain activity from 26 healthy young subjects (14 females) related to shallow perceptual and deep semantic word encoding. During subsequent recognition tests, the subjects had to recognize the previously encoded words which were interspersed with new words. The resulting mean memory performances across all subjects clearly mirrored the different levels of encoding. The grand averaged event-related fields (ERFs) associated with perceptual and semantic word encoding differed significantly between 200 and 550 ms after stimulus onset mainly over left superior temporal and left superior parietal sensors. Semantic encoding elicited higher brain activity than perceptual encoding. Source localization procedures revealed that neural populations of the left temporal and temporoparietal brain areas showed different activity strengths across the whole group of subjects depending on depth of word encoding. We suggest that the higher brain activity associated with deep encoding as compared to shallow encoding was due to the involvement of more neural systems during the processing of visually presented words. Deep encoding required more energy than shallow encoding but for all that led to a better memory performance.     

Modulation of the lexical-semantic network by auditory semantic priming: an event-related functional MRI study

The current event-related fMRI study specifies the neuroanatomical correlates of semantic priming and differences in semantic relation types using an auditory primed lexical decision task (LDT). Word pairs consisted of different relation types, associations (key-chain), pure categorical relations (cow-dog), and unrelated words (table-window), as well as word-pseudoword (way-tinne) and pseudoword-pseudoword (ahurn-d?va) pairs. The factor lexical status, i.e., the processing of words compared to pseudowords, was associated with activation in the middle temporal gyri and the left striatum. The factor relatedness, i.e., the contrast between unrelated and related target words, was associated with increased activation of the left inferior frontal gyrus, the deep frontal operculum bilaterally, and the middle frontal gyri. A direct contrast between the two semantic relation types indicated that the processing of purely categorical compared to associative information recruits the right precuneus, the isthmus gyrus cinguli, and the cuneus, suggesting more effortful processing of the former information type. The present data show that the factors lexical status, semantic relatedness, and type of semantic relation in a primed LDT modulate the hemodynamic response in cerebral areas that subserve auditory word recognition and subsequent lexical-semantic processing.     

The FN400 indexes familiarity-based recognition of faces

Separate event-related brain potential (ERP) components have been hypothesized to index familiarity and recollection processes that support recognition memory. A 300- to 500-ms mid-frontal FN400 old/new difference has been related to familiarity, whereas a 500- to 800-ms parietal old/new difference has been related to recollection. Other recent work has cast doubt on the FN400 familiarity hypothesis, especially its application to familiarity-based recognition of conceptually impoverished stimuli such as novel faces. Here we show that FN400 old/new differences can be observed with novel faces, and as predicted by the familiarity hypothesis, these differences are observed regardless of whether or not recognition is accompanied by the recollection of specific details from the study episode. Furthermore, FN400 differentiation between hits and misses is more consistent with an explicit familiarity process than an implicit memory process.     

Encoding activity in anterior medial temporal lobe supports subsequent associative recognition

The ability to bind information together, such as linking a name with a face or a car with a parking space, is a vital process in human episodic memory. To identify the neural bases for this binding process, we measured brain activity during a verbal associative encoding task using event-related functional MRI (fMRI), followed by an associative recognition test for the studied word pairs. Analysis of the encoding data sorted by the associative recognition accuracy allowed us to isolate regions involved in successfully creating associations. We found that encoding activity in bilateral anterior medial temporal lobe (MTL) regions was greater for successfully bound pairs, that is, those later recognized as intact, than for all other pairs. These findings provide evidence that the anterior medial temporal lobes support the successful binding of information in memory.     

Juicy fruit and creepy crawlies: an electrophysiological study of the implicit Go/NoGo association task

The Go/NoGo association task (GNAT) has been used in behavioral studies to measure the strength of association between different category groups and two poles of an evaluative dimension. However, reaction time data do not provide information about the neural time course of such associative information. We investigated event-related brain potentials (ERPs) elicited when participants were required to respond (Go) or withhold a response (NoGo) according to task instructions. Task instructions paired words from one of two taxonomic categories (fruit/bugs) with either pole of an evaluative dimension (good/bad). Within a given run, Go responses were assigned to one of the categories and one evaluative dimension. ERPs showed an increased negativity over frontal sites to NoGo as compared to Go responses. Moreover, NoGo minus Go difference waves showed that the N200 effect was delayed in trials within incongruent blocks (e.g., "Press if a bug word or a good word") as compared to trials within congruent blocks (e.g., "Press if a bug word or a bad word"). These results suggest that such associative attitude information is available at a very early stage of processing, less than 250 ms after seeing a fruit or a bug word. This finding is further discussed with respect to alternative explanations of the behavioral effect.     

"Sculpting the response space"--an account of left prefrontal activation at encoding

Left lateral prefrontal cortex (PFC) is consistently activated in neuroimaging studies of memory encoding. Its role, however, remains unclear. We describe two functional magnetic resonance imaging (fMRI) studies addressing this question. In the first we used a blocked experimental design to explore the effect of repeated encoding of word paired associates. Initial presentation of word pairs was associated with left ventrolateral PFC activation that attenuated with subsequent presentations of the same lists. When well-learned lists were presented with word pairs rearranged, a left PFC activation, greater than that associated with the initial presentation, was observed. In a second experiment, the formation of these associative relationships was explored using an event-related design. Two types of word pairs were presented: closely related (e.g., King...Queen) and distantly related (e.g., Net...Ship). The same region of left PFC was differentially sensitive to these two event-types, showing a greater response for distantly related pairs. We suggest that left PFC activity, at memory encoding, reflects operations necessary to the formation of meaningful associations in the service of optimal learning. A crucial feature of such associative processing lies in selecting appropriate, and inhibiting inappropriate, semantic attributes of the study material.     

True and phantom recollection: an fMRI investigation of similar and distinct neural correlates and connectivity

Although research suggests that most false memories are mediated by a sense of familiarity, behavioral evidence indicates that some are characterized by retrieval of item-specific details associated with recollection. However, neuroimaging studies have yet to isolate and analyze the neural correlates of false (or phantom) recollection, focusing instead on general recognition processes. In doing so, results are mixed with respect to the role of the medial temporal lobes (MTL) in distinguishing between true and false retrieval. The present study sought to investigate the neural basis of true and phantom recollection and clarify the role of the MTL in dissociating between the two processes. Results showed that true and phantom recollection were associated with a largely overlapping retrieval network including activity in bilateral anterior parahippocampal gyrus, fusiform gyrus, anterior cingulate cortex, and right superior parietal cortex. However, connectivity analyses using two common MTL seeds revealed a more inferior network (fusiform gyrus, hippocampus, middle temporal gyrus) associated with true recollection and a more superior network (superior parietal, superior frontal gyrus, posterior cingulate cortex) associated with false recollection. Finally, direct comparisons between true and phantom recollection showed greater activity in right hippocampus and early visual cortex for true recollection, whereas no region exhibited greater activity for false recollection. Results indicate that while both true and phantom recollection show similar patterns of activation, there are also distinctions in the neural networks contributing to the two recollection processes. Moreover, results conclude that within the MTL, the hippocampus proper can distinguish between true and phantom recollection.     

Effects of task-set adoption on ERP correlates of controlled and automatic recognition memory

Successful memory retrieval depends not only on memory fidelity but also on the mental preparedness on the part of the subject. ERP studies of recognition memory have identified two topographically distinct ERP components, the FN400 old/new effect and the late posterior component (LPC) old/new effect, commonly associated with familiarity and recollection, respectively. Here we used a task-switching paradigm to examine the extent to which adoption of a retrieval task-set influences FN400 and LPC old/new effects, in light of the presumption that recollection, as a control process, relies on the adoption of a retrieval task-set, but that familiarity-based retrieval does not. Behavioral accuracy indicated that source memory (experiment 2), but not item recognition (experiment 1), improved with task-set adoption. ERP data demonstrated a larger LPC on stay trials when a task-set had been adopted even with a simple recognition memory judgment. We conclude that adopting a retrieval task-set impacts recollection memory but not familiarity. These data indicate that attentional state immediately prior to retrieval can influence objective measures of recollection memory.