Brain activity from stimuli that are not perceived: Visual mismatch negativity during binocular rivalry suppression

Predictive coding explains visual perception as the result of an interaction between bottom‐up sensory input and top‐down generative models at each level of the visual hierarchy. Evidence for this comes from the visual mismatch negativity (vMMN): a more negative ERP for rare, unpredictable visual stimuli—deviants, than for frequent, predictable visual stimuli—standards. Here, we show that the vMMN does not require conscious experience. We measured the vMMN from monocular luminance‐decrement deviants that were either perceived or not during binocular rivalry dominance or suppression, respectively. We found that both sorts of deviants elicited the vMMN at about 250 ms after stimulus onset, with perceived deviants eliciting a bigger vMMN than not‐perceived deviants. These results show that vMMN occurs in the absence of consciousness, and that consciousness enhances the processing underlying vMMN. We conclude that generative models of visual perception are tested, even when sensory input for those models is not perceived.     

The human brain processes visual changes that are not cued by attended auditory stimulation

Event-related potentials (ERPs) to visual stimuli were recorded from the scalp of eight adult humans performing a task in which they counted vowels from a heard story. In the oddball condition, a repeated (standard) light bar of 50 ms in duration was rarely (P = 0.1) replaced by a (deviant) one differing in orientation from the standard. In the control condition, standards were simply omitted from the series and only (alone-) deviants retained. In both conditions, visual stimuli were asynchronous with auditory-task-relevant stimuli. ERPs to deviants significantly differed in amplitude from those to standards in the midline electrodes centrally, parietally and occipitally at 160-200 ms from stimulus onset. Occipitally, such a difference was absent between ERPs to alone-deviants and those to standards. The occipital differential ERPs to deviants, which thus could be found only when standards were present in the series, are discussed in the context of the mismatch negativity (MMN).     

Magnetic Source Localization of Early Visual Mismatch Response

Previous studies have reported a visual analogue of the auditory mismatch negativity (MMN) response that is based on sensory memory. The neural generators and attention dependence of the visual MMN (vMMN) still remain unclear. We used magnetoencephalography (MEG) and spatio-temporal source localization to determine the generators of the sensory-memory-based vMMN response to non-attended deviants. Ten participants were asked to discriminate between odd and even digits presented at the center of the visual field while grating patterns with different spatial frequencies were presented outside the focus of attention. vMMN was calculated as the difference between MEG responses to infrequent gratings in oddball blocks and the same gratings in equiprobable blocks. The peak latency of the vMMN response was between 100 and 160 ms. The neuromagnetic sources of the vMMN localized in the occipital cortex differed from the sources evoked by the equiprobable gratings and were stimulus-dependent. Our results suggest the existence of separate neural systems for pre-attentive memory-based detection of visual change and provide new evidence that the vMMN is feature-specific.     

Impairment in processing visual information at the pre-attentive stage in patients with a major depressive disorder: A visual mismatch negativity study

We compared the pre-attentive processing in patients with a major depressive disorder (MDD) and matched healthy controls as indexed by the visual mismatch negativity (vMMN) elicited by exposure duration of visual stimuli randomly presented on both peripheral visual fields. To obtain the memory-comparison-based visual MMN, the role of standard and deviant stimuli was reversed in separate blocks. Compared with healthy participants, MDD patients exhibited decreased MMN amplitudes of long duration deviant only and this deficit was not correlated with the depression severity. These data suggests functional impairment of pre-attentive basic visual information processing in MDD patients.     

Can Echoic Memory Store Two Traces Simultaneously? A Study of Event-Related Brain Potentials

The mismatch negativity, a component of the event-related brain potential elicited by infrequent deviants in sequences of auditory stimuli, is presumably generated by an automatic mismatch process in a mechanism that compares the current stimulus to the trace of the previous one. The present study addressed the possible simultaneous existence of two such traces. Two equiprobable (45% each) frequent stimuli (“standards”), one of 600 Hz and the other of 700 Hz, were presented together with an infrequent (10%), “deviant” stimulus which was of differnet frequency in different blocks. These deviants elicited a mismatch negativity, though a smaller one than that obtained in corresponding blocks with only one standard stimulus. Two aspects of the present results from the blocks with two standard stimuli implicate two parallel stimulus traces in these blocks: 1) deviants elicited a mismatch negativity (MMN) of approximately the same amplitude when preceded by sequences of four identical standards as when preceded by sequences of four stimuli containing both standards; 2) in contrast to the one-standard condition, the magnitude of stimulus deviance did not affect the MMN component elicited by the different deviants.     

Backward masking and visual mismatch negativity: electrophysiological evidence for memory-based detection of deviant stimuli

Sequences composed of two different colored checkerboard patterns (standard and deviant) were presented to adults. Each pattern was followed by a mask with stimulus onset asynchronies (SOAs) varying between 14 and 174 ms. ERPs were recorded to the deviant and standard stimuli while the participants detected changes of a cross, which was continuously present at the center of the screen. In further experiments, the participants performed a Go-NoGo task detecting the deviant checkerboards. Deviant stimuli elicited an occipital negative component with 124-132 ms mean latency (the visual mismatch negativity, vMMN) at test (standard or deviant)-to-mask SOAs longer than 27 ms. No vMMN amplitude increase was observed beyond 40 ms test-to-mask intervals, whereas detection of deviant checkerboard patterns improved up to 174-ms SOA. Therefore the processes underlying vMMN elicitation cannot fully explain the overt detection of visual deviance.     

Deviance detection in congruent audiovisual speech: Evidence for implicit integrated audiovisual memory representations

Detection of deviant speech syllables embedded in continuous noise was investigated in an oddball paradigm. Behavioral results showed improvement of detecting and identifying the syllables when congruent visual speech accompanied the utterances. A centrally maximal negative ERP difference wave peaking at approximately 290 ms post-stimulus was elicited by audiovisual but not by auditory- or visual-only task-irrelevant deviant syllables. Whereas the circumstances of the elicitation of this ERP response are similar to those of the mismatch negativity component (MMN and its visual counterpart, vMMN), its scalp distribution differs from that of both unimodal MMNs. Elicitation of an MMN-like ERP response (termed here as the audiovisual MMN: avMMN) suggests that detection of the audiovisual deviants involved integrated audiovisual memory representations. The pattern of behavioral and ERP results suggest that the formation of such cross-modal memory representation does not require voluntary operations and may even proceed for stimuli outside the focus of attention.     

Short‐term trained lexical categories produce preattentive categorical perception of color: Evidence from ERPs

The present study investigated whether short‐term trained lexical categories could produce lateralized preattentive categorical perception (CP) of color. Participants' event‐related potentials were recorded while performing a visual oddball task in which standard and deviant colored stimuli from the same or different novel lexical categories were presented. Two groups of participants were recruited: a group trained on these novel categories (n = 26), and an untrained control group (n = 26). Results of paired t tests showed that deviants did not evoke significant visual mismatch negativity, with the exception of deviants from different novel categories presented in the right visual field of the training group. This suggests that short‐term trained lexical categories produce lateralized preattentive color CP, and language enhances sensitivity to the differences among between‐category stimuli.     

Changes in Evoked Potentials during the Action of Sound Signals with Different Localizing Characteristics

This report presents results of studies of the phenomenon of mismatch negativity (MMN) during exposure to four blocks of sound stimuli each containing identical standards creating an immobile sound image located along the midline of the head and one of a set of deviants, creating a sound image located either by the left ear or moving from the midline of the head towards the left ear or in the opposite direction. All deviants induced mismatch negativity; the minimal amplitude and longest latent period were seen in the mismatch negativity produced by the deviant modeling movement of the sound image from the midline of the head to the left ear. The question of the appearance of mismatch negativity as a criterion for the accurate discrimination of signals with different localizing characteristics is discussed.     

Dysfunction of processing task-irrelevant emotional faces in major depressive disorder patients revealed by expression-related visual MMN

We compared the pre-attentive processing of emotional faces in major depressive disorder (MDD) patients and matched healthy controls as indexed by the visual mismatch negativity (vMMN) elicited by emotional faces presented in upright and inverted orientations. Compared with standard neutral faces, infrequent emotional faces elicited posterior negativities between 100 ms and 350 ms. Early vMMN was smaller, while late vMMN was absent in the MDD group, as compared with the normal group. Face inversion decreased the vMMN in the normal group but had no effect on MDD patients. These data suggest the functional dysfunction of pre-attentive expressional information processing in MDD patients, coupled with the disorder of high-level perceptual strategy of processing emotional faces.     

Color categories affect pre-attentive color perception

Categorical perception (CP) of color is the faster and/or more accurate discrimination of colors from different categories than equivalently spaced colors from the same category. Here, we investigate whether color CP at early stages of chromatic processing is independent of top-down modulation from attention. A visual oddball task was employed where frequent and infrequent colored stimuli were either same- or different-category, with chromatic differences equated across conditions. Stimuli were presented peripheral to a central distractor task to elicit an event-related potential (ERP) known as the visual mismatch negativity (vMMN). The vMMN is an index of automatic and pre-attentive visual change detection arising from generating loci in visual cortices. The results revealed a greater vMMN for different-category than same-category change detection when stimuli appeared in the lower visual field, and an absence of attention-related ERP components. The findings provide the first clear evidence for an automatic and pre-attentive categorical code for color.     

Is the auditory sensory memory sensitive to visual information?

The mismatch negativity (MMN) component of auditory event-related brain potentials can be used as a probe to study the representation of sounds in auditory sensory memory (ASM). Yet it has been shown that an auditory MMN can also be elicited by an illusory auditory deviance induced by visual changes. This suggests that some visual information may be encoded in ASM and is accessible to the auditory MMN process. It is not known, however, whether visual information affects ASM representation for any audiovisual event or whether this phenomenon is limited to specific domains in which strong audiovisual illusions occur. To highlight this issue, we have compared the topographies of MMNs elicited by non-speech audiovisual stimuli deviating from audiovisual standards on the visual, the auditory, or both dimensions. Contrary to what occurs with audiovisual illusions, each unimodal deviant elicited sensory-specific MMNs, and the MMN to audiovisual deviants included both sensory components. The visual MMN was, however, different from a genuine visual MMN obtained in a visual-only control oddball paradigm, suggesting that auditory and visual information interacts before the MMN process occurs. Furthermore, the MMN to audiovisual deviants was significantly different from the sum of the two sensory-specific MMNs, showing that the processes of visual and auditory change detection are not completely independent.     

Immediate integration of prosodic information from speech and visual information from pictures in the absence of focused attention: A mismatch negativity study

Language is often perceived together with visual information. Recent experimental evidences indicated that, during spoken language comprehension, the brain can immediately integrate visual information with semantic or syntactic information from speech. Here we used the mismatch negativity to further investigate whether prosodic information from speech could be immediately integrated into a visual scene context or not, and especially the time course and automaticity of this integration process. Sixteen Chinese native speakers participated in the study. The materials included Chinese spoken sentences and picture pairs. In the audiovisual situation, relative to the concomitant pictures, the spoken sentence was appropriately accented in the standard stimuli, but inappropriately accented in the two kinds of deviant stimuli. In the purely auditory situation, the speech sentences were presented without pictures. It was found that the deviants evoked mismatch responses in both audiovisual and purely auditory situations; the mismatch negativity in the purely auditory situation peaked at the same time as, but was weaker than that evoked by the same deviant speech sounds in the audiovisual situation. This pattern of results suggested immediate integration of prosodic information from speech and visual information from pictures in the absence of focused attention.     

Visual change detection: event-related potentials are dependent on stimulus location in humans

Infrequent colored patterns within sequences of patterns of frequent color elicited a posterior negative event-related potential component only in case of lower half-field stimulation. This negativity in the 140-200 ms latency range is considered as a correlate of automatic visual change detection (visual mismatch negativity, vMMN). Retinotopic prestriate visual areas are suggested to be the generating loci of vMMN.     

Unnoticed regularity violation elicits change-related brain activity

Event-related brain electric activity (ERP) was investigated to unnoticed visual changes. The orientation of grid elements (vertical or horizontal) changed after the presentation of 10–15 identical stimuli. The grid patterns were task irrelevant, and were presented in the background of a shape discrimination task. During the first half of the session, participants were unaware of the stimulus change. However, in comparison to the ERPs to the fifth identical stimuli, stimulus change elicited posterior negativities in the 270–375 ms range (visual mismatch negativity, vMMN). With participants instructed on the stimulus change, negativities emerged with earlier onset and with wider distribution. When stimulus change was preceded by only two identical stimuli, there were no such ERP effects. As the results show, a longer sequence of identical unattended stimuli may establish the memory representation of stimulus regularity, and violation of regularity is indicated by posterior negative ERP components (vMMN).     

Regularity encoding and deviance detection of frequency modulated sweeps: Human middle‐ and long‐latency auditory evoked potentials

Fast encoding of frequency modulated (FM) sweeps is crucial for communication. In humans, FM sweeps deviating from the acoustic regularity elicit the mismatch negativity (MMN) evoked potential. Yet, direction sensitivity to FM sweeps is found in animals' primary auditory cortex, upstream of MMN sources found in humans. Here, we were interested in whether direction deviants of complex FM sweeps modulated brain responses earlier than MMN. We used a controlled oddball paradigm, and measured the middle latency responses (MLRs) and the MMN. Our results showed a repetition enhancement to the standards at the Pa component of the MLR and a genuine MMN in the later response range. These results show that, early in the cortical hierarchy, the system is sensitive to the physical characteristics of the repetitive stimuli, but a higher‐order mechanism is needed to detect violations of the acoustic regularity.     

ERPs and deviance detection: visual mismatch negativity to repeated visual stimuli

Previous studies showed a visual analogue of the mismatch negativity (MMN) event-related potential (ERP), which is elicited by violating some sensory regularity. Separating physical change from violating a regularity, here we show that the visual MMN (vMMN) is elicited by regularity violations that do not involve physical stimulus change. Adult participants were presented with a series of red-black and green-black checkerboard patterns delivered regularly in an RRGGRRGG ... order. Infrequently (p=0.1) this regularity was broken by repeating a stimulus one additional time (e.g. RRGGRRR). ERPs elicited by irregular stimulus repetitions were negatively displaced compared to those elicited both by regular repetition and regular change in two latency ranges: 100-140 ms and 220-260 ms. Whereas the first of these two negative ERP differences appears to be sensitive to stimulus repetition per se, the second difference can be identified as a vMMN response to violating the sequential regularity. Thus, similarly to its auditory counterpart, vMMN reflects deviance-(regularity-violation) rather than change-detection processes.     

Responses of the human auditory cortex to changes in one versus two stimulus features

Neuromagnetic responses were recorded with a 24 SQUID magnetometer in two oddball experiments to determine whether mismatch responses to changes in single stimulus features are additive. In experiment 1, the one feature deviants differed from standards in interstimulus interval (ISI) or frequency, and the two feature deviants in both ISI and frequence. In experiment 2, deviants differed in duration, frequency, or both. All deviants evoked a mismatch field (MMF) with sources close to each other in the supratemporal auditory cortex. Except for the ISI deviants, the MMF sources were about 1 cm anterior to the source of the 100ms response, N100m, to the standards. In the two experiments, MMFs obtained in response to the two feature deviants resembled closely the sum of MMFs in response to one feature deviants. The results suggest that the standards leave a multiple neuronal representation in the human auditory cortex. The particular neuronal traces of the representation react independently to changes in different features of sound stimuli.     

Attention and mismatch negativity

The mismatch negativity (MMN) component of the auditory event-related potential (ERP) is elicited by infrequent, physically deviant stimuli in a sequence of frequent homogeneous stimuli (standards). It has been suggested that the MMN is generated by an automatic (attention-independent) neural mismatch process with a memory trace that encodes the physical features of the standard stimulus. The proposed MMN independence of attention was addressed in the present study. Standard stimuli and two types of deviant stimuli, differing from standards either in frequency or intensity, were dichotically presented in random order and at a very rapid rate. The subject attended either to left-or right-ear stimuli, counting the number of a designated type of deviants in that ear. In the present conditions of very strongly focused attention, the MMN was elicited even by frequency change in the ignored input stream, and its amplitude was very similar to that of the MMN elicited by equivalent deviant stimuli (targets) in the attended input stream. In contrast, the MMN to intensity deviation was clearly attenuated in the absence of attention. This effect is, however, probably due to the attention effect on the MMN generator itself rather than the antecedent sensory-analysis and -storing functions.     

Tonotopic auditory cortex and the magnetoencephalographic (MEG) equivalent of the mismatch negativity

Two tone stimuli, one frequent (standard) and the other infrequent (a slightly higher, deviant tone), were presented in random order and at short intervals to subjects reading texts they had selected. In different blocks, standards were either 250, 1,000, or 4,000 Hz, with the deviants always being 10% higher in frequency than the standards of the same blocks. Magnetic responses elicited by the standard and deviant tones included N1m, the magnetoencephalographic equivalent of the electrical N1 (its supratemporal component). In addition, deviant stimuli elicited MMNm, the magnetic equivalent of the electrical mismatch negativity, MMN. The equivalent dipole sources of the two responses were located in supratemporal auditory cortex, with the MMNm source being anterior to that of N1m. The dipole orientations of both sources in the sagittal plane depended on stimulus frequency, suggesting that the responses are generated by tonotopically organized neuronal populations. The tonotopy reflected by the frequency dependence of the MMNm source might be that of the neural trace system underlying frequency representation of auditory stimuli in sensory memory.