Effects of temporal encoding on auditory object formation: a mismatch negativity study

Analysis of auditory scene relies on the appropriate division of the sensory input into meaningful events. The auditory system uses a sequential grouping strategy to segregate different acoustic events in the time domain. As a result of grouping, the organization of sound is expected to include acoustic events spanning different scales of time. It is unclear, however, at which moment the central auditory system begins to introduce changes in the neural representation of the auditory scene as a result of grouping different information units into a larger acoustic event. The present set of experiments examines this issue by using the mismatch negativity (MMN) component of event-related potentials. The MMN is elicited in response to changes in the previously registered auditory regularity whether or not the subject's attention is focused on the auditory stimuli. By presenting three consecutive pairs of alternating tones in serial stimulus trains, we found that across-train repetitions of the tone elicited the MMN with inter-train intervals up to 240 ms. Beyond this interval, only within-train repetitions of the tone elicited the MMN, suggesting that pairs of tones were bound together into a single acoustic event (i.e., train of six tones) within a temporal window not much longer than 240 ms. Furthermore, a different pattern of responses was seen depending on the position of the deviant event within the train and the frequency distance between the alternating tones. These results suggest that grouping discrete auditory inputs may introduce changes in the neural representation of the auditory scene by applying rules to sounds spanning larger temporal scales.     

Spectrotemporal window of integration of auditory information in the human brain

The human auditory system is adapted to integrate temporally successive sounds into meaningful entities, that is, acoustic information units. Hence, sound sequences falling within the temporal window of integration should be coded holistically as unitary representations in the human auditory cortex. Although it is well established that the auditory system operates in the frequency-temporal domain, many previous studies only focused on the temporal domain of the window of integration. Therefore, in the current study we investigated the relationship between the short-term temporal integration and the frequency integration. Event-related magnetic fields in response to infrequent omission of the second tone in repetitive tone pairs composed of two closely spaced tones of different frequencies were recorded. This omission elicited the magnetic counterpart (MMNm) of the electric mismatch negativity (MMN), a change-specific component mainly generated in the auditory cortex, when the interval between the two successive tones was extremely short or when the frequency difference between the two tones was small. These findings suggest that two stimuli presented in close succession might be represented in the auditory system as a unitary integrated event. In addition, as the distance between the two successive tones decreased in the spectrotemporal dimensions, the magnitude of the MMNm increased. Behavioral data also supported these neurophysiological phenomena. This work shows the first neurophysiological evidence that the two-dimensional (spectrotemporal) window of integration, which provides important constraints for the neural processing of the acoustic environment, exists in the human brain.     

Simultaneously active pre-attentive representations of local and global rules for sound sequences in the human brain

Regular sequences of sounds (i.e., non-random) can usually be described by several, equally valid rules. Rules allowing extrapolation from one sound to the next are termed local rules, those that define relations between temporally non-adjacent sounds are termed global rules. The aim of the present study was to determine whether both local and global rules can be simultaneously extracted from a sound sequence even when attention is directed away from the auditory stimuli. The pre-attentive representation of a sequence of two alternating tones (differing only in frequency) was investigated using the mismatch negativity (MMN) auditory event-related potential. Both local- and global-rule violations of tone alternation elicited the MMN component while subjects ignored the auditory stimuli. This finding suggests that (a) pre-attentive auditory processes can extract both local and global rules from sound sequences, and (b) that several regularity representations of a sound sequence are simultaneously maintained during the pre-attentive phase of auditory stimulus processing.     

Neural representations of auditory input accommodate to the context in a dynamically changing acoustic environment

The auditory scene is dynamic, changing from 1 min to the next as sound sources enter and leave our space. How does the brain resolve the problem of maintaining neural representations of the distinct yet changing sound sources? We used an auditory streaming paradigm to test the dynamics of multiple sound source representation, when switching between integrated and segregated sound streams. The mismatch negativity (MMN) component of event‐related potentials was used as index of change detection to observe stimulus‐driven modulation of the ongoing sound organization. Probe tones were presented randomly within ambiguously organized sound sequences to reveal whether the neurophysiological representation of the sounds was integrated (no MMN) or segregated (MMN). The pattern of results demonstrated context‐dependent responses to a single tone that was modulated in dynamic fashion as the auditory environment rapidly changed from integrated to segregated sounds. This suggests a rapid form of auditory plasticity in which the longer‐term sound context influences the current state of neural activity when it is ambiguous. These results demonstrate stimulus‐driven modulation of neural activity that accommodates to the dynamically changing acoustic environment.     

Optical imaging of temporal integration in human auditory cortex

Behavioral and physiological studies have indicated the existence of a temporal window of auditory integration (TWI), within which similar sounds are perceptually grouped. The current study exploits the combined temporal and spatial resolution of fast optical imaging (the event-related optical signal, EROS) to show that brain activity elicited by sounds within and outside the TWI differs in location and latency. In a previous event-related brain potential (ERP) study [Sable, Gratton, and Fabiani (2003) European Journal of Neuroscience, 17, 2492-2496], we found that the mismatch negativity (MMN; a brain response to acoustic irregularities) elicited by deviations in stimulus onset asynchronies (SOAs) had a unique shape when the deviant SOA was within the TWI. In the present study, we extended these ERP results using EROS. Participants heard trains of five tones. The first four tones had SOAs of 96, 192, 288 or 384 ms. The SOA of the fourth and fifth tones was either the same (standard) or one of the other three (deviant) SOAs. With a deviant SOA of 96 ms, the cortical response was approximately 2 cm anterior to responses to longer SOA deviants, and was followed by a later response that was absent in the other conditions. Similarly to the electrical MMN, the optical mismatch response amplitudes were proportional to the magnitude of interval deviance. These results, in combination with our previous findings, indicate that the temporal integration of sounds is reflected in cortical mismatch responses that differ from the typical response to interval deviance.     

Effects of attention on neuroelectric correlates of auditory stream segregation

A general assumption underlying auditory scene analysis is that the initial grouping of acoustic elements is independent of attention. The effects of attention on auditory stream segregation were investigated by recording event-related potentials (ERPs) while participants either attended to sound stimuli and indicated whether they heard one or two streams or watched a muted movie. The stimuli were pure-tone ABA--patterns that repeated for 10.8 sec with a stimulus onset asynchrony between A and B tones of 100 msec in which the A tone was fixed at 500 Hz, the B tone could be 500, 625, 750, or 1000 Hz, and--was a silence. In both listening conditions, an enhancement of the auditory-evoked response (P1-N1-P2 and N1c) to the B tone varied with Deltaf and correlated with perception of streaming. The ERP from 150 to 250 msec after the beginning of the repeating ABA- patterns became more positive during the course of the trial and was diminished when participants ignored the tones, consistent with behavioral studies indicating that streaming takes several seconds to build up. The N1c enhancement and the buildup over time were larger at right than left temporal electrodes, suggesting a right-hemisphere dominance for stream segregation. Sources in Heschl's gyrus accounted for the ERP modulations related to Deltaf-based segregation and buildup. These findings provide evidence for two cortical mechanisms of streaming: automatic segregation of sounds and attention-dependent buildup process that integrates successive tones within streams over several seconds.     

Automatic discriminative sensitivity inside temporal window of sensory memory as a function of time

Neural representation of preceding sound-patterns stored in the human brain, as reflected by mismatch negativity (MMN) related to the automatic discriminative process, is restricted to a duration of 160-170 ms due to the short form of auditory sensory memory termed the temporal window of integration (TWI). To examine the temporal uniformity of deviation-sensitivity inside TWI of sensory memory, magnetic MMN (MMNm) responses were measured with a dual 37-channel magnetometer for complex sounds of 170 ms duration containing an omitted (silent) segment. Frequent standard stimuli (probability of 80%) consisted of five tone segments. Deviant stimuli were different from standard stimuli in that one of four segments was occasionally (probability of 5%) omitted and replaced by a silent segment. The stimulus duration of 170 ms was intended to correspond to the postulated duration of TWI. When the silent segment occurred later in deviant stimulus, the MMNm peak amplitude was attenuated and MMNm peak latency, measured from the onset of each silent segment, was delayed. Thus, automatic deviation-detection sensitivity declines nonlinearly toward the end of TWI in auditory sensory memory. In the second experiment, two types of deviant stimuli, which differed from each other only in the period after the occurrence of the silent segment, elicited MMNm with the same peak latency but with a different peak amplitude. Thus, mismatch process is triggered at the moment of change but still lasts after the detection of deviation. In other words, both standard and deviant stimuli are treated as a unitary event within a TWI.     

Temporal resolution properties of human auditory cortex: reflections in the neuromagnetic auditory evoked M100 component

Previous work has provided evidence for a brief, finite ( approximately 35 ms) temporal window of integration (TWI) in M100 formation, during which stimulus attributes are accumulated in processes leading to the M100 peak. Here, we investigate resolution within the TWI by recording responses to tones containing silent gaps (0-20 ms). Gaps were inserted in 1 kHz tones in 2 conditions: +10 ms post-onset (10 ms masker) wherein the masker and gap of longest duration (20 ms) were contained within the initial 35 ms of the stimulus and +40 ms (40 ms masker) wherein all gaps were inserted +40 ms post-onset. Tones were presented binaurally and responses sampled from both hemispheres in 12 adults using a twin 37-channel biomagnetometer (MAGNES-II, BTi, San Diego, CA). Results--10 ms masker: M100 latency was prolonged and amplitude decreased as a function of gap duration, even with the shortest duration (2 ms) gap, indicating that integrative processes underlying M100 formation are sensitive to fine-grained discontinuities within a brief, finite TWI. Results--40 ms masker: M100 latency and amplitude were unaffected by gaps inserted at +40 ms, providing further evidence for an M100 TWI of <40 ms. CONCLUSION: within a brief integrative window in M100 formation, population-level responses are sensitive to discontinuities in sounds on a scale corresponding to psychophysical detection thresholds and minimum detectable gap thresholds in single unit recordings. Cumulatively, results provide evidence that M100 resolution for brief fluctuations in sounds reflects temporal acuity properties that are both intrinsic to the auditory system and critical to the accurate perception of speech.     

Auditory stream segregation in the songbird forebrain: effects of time intervals on responses to interleaved tone sequences

For both humans and other animals, the abilities to integrate separate sound elements over time into coherent perceptual representations, or 'auditory streams', and to segregate these auditory streams from other interleaved sounds are critical for hearing and vocal communication. In humans and European starlings (Sturnus vulgaris) the ability to perceptually segregate a simple interleaved tone sequence comprised of two alternating tones differing in frequency (ABA-ABA-ABA-...) into separate auditory streams of A and B tones is promoted at larger frequency separations (DeltaF) between the A and B tones. In humans, segregating A and B tones into different streams also appears to be promoted at shorter interstimulus intervals (ISI) between tones within a stream (e.g., between consecutive A tones). Here, we used the ABA experimental paradigm to investigate the influence of different time intervals between A and B tones in repeated ABA triplets on neural responses in the starling forebrain. The main finding from the study is that a DeltaF-dependent effect of ISI had a large influence on the relative responses to A and B tones. Responses to B tones were suppressed, relative to A-tone responses, when the A and B tones were more similar in frequency (smaller DeltaFs) and occurred at shorter ISIs. We attribute these suppressive effects to physiological forward masking and suggest that forward masking functions as a mechanism for segregating neural responses to interleaved tones in tonotopic space. We discuss the relevance of our physiological data with respect to previous electrophysiological studies of auditory stream segregation in mammals and previous perceptual studies in humans.     

The individual replicability of mismatch negativity at short and long inter-stimulus intervals.

OBJECTIVES: The individual replicability of the mismatch negativity (MMN) event-related brain potential (ERP) was studied at two different inter-stimulus intervals (ISIs), to establish its potential value for routine clinical evaluation of sound discrimination and auditory sensory memory. METHODS: Ten healthy young subjects were presented sequences of 3 stimulus trains, in two recording sessions approximately 1 month apart. The stimuli in the trains were delivered at an ISI of 300 ms, whereas the inter-train intervals (ITIs) were 0.4 s and 4.0 s in different blocks. ERPs were averaged to standard (75 ms) and deviant (25 ms) tones started equiprobably the stimulus trains. RESULTS: Significant Pearson product-moment correlations coefficients were found between sessions at all scalp locations for the short ITI, when the MMN was quantified as the mean amplitude in the 100-200 ms latency window around its peak. However, none of the correlations reached significance for the longer ITI. CONCLUSIONS: MMN appears to be a reliable measure for single-case assessment and follow-ups when obtained at short ISIs and quantified as an integrated window of neuroelectric activation over a temporal span.     

Temporal integration: intentional sound discrimination does not modulate stimulus-driven processes in auditory event synthesis

Objective: Our previous study showed that the auditory context could influence whether two successive acoustic changes occurring within the temporal integration window (approximately 200 ms) were pre-attentively encoded as a single auditory event or as two discrete events (Cogn Brain Res 12 (2001) 431). The aim of the current study was to assess whether top-down processes could influence the stimulus-driven processes in determining what constitutes an auditory event.Methods: Electroencepholagram (EEG) was recorded from 11 scalp electrodes to frequently occurring standard and infrequently occurring deviant sounds. Within the stimulus blocks, deviants either occurred only in pairs (successive feature changes) or both singly and in pairs. Event-related potential indices of change and target detection, the mismatch negativity (MMN) and the N2b component, respectively, were compared with the simultaneously measured performance in discriminating the deviants.Results: Even though subjects could voluntarily distinguish the two successive auditory feature changes from each other, which was also indicated by the elicitation of the N2b target-detection response, top-down processes did not modify the event organization reflected by the MMN response.Conclusions: Top-down processes can extract elemental auditory information from a single integrated acoustic event, but the extraction occurs at a later processing stage than the one whose outcome is indexed by MMN.Significance: Initial processes of auditory event-formation are fully governed by the context within which the sounds occur. Perception of the deviants as two separate sound events (the top-down effects) did not change the initial neural representation of the same deviants as one event (indexed by the MMN), without a corresponding change in the stimulus-driven sound organization.     

The development of the length of the temporal window of integration for rapidly presented auditory information as indexed by MMN.

OBJECTIVE: The length of the temporal window integrating successive auditory stimuli into unitary percepts has been estimated to be less than 200 ms in adults. The aim of the study was to investigate the development of the integrating window in children to determine whether it is similar to young adults. METHODS: A modified auditory oddball paradigm was presented at a rapid (150-400 ms) onset-to-onset pace during recording of electroencephalogram in two groups of children (aged 5-8 and 9-11 years) and one group of adults. Latencies and amplitudes of the P1, N1, and the mismatch negativity (MMN) components of event-related brain potentials (ERPs) were measured. RESULTS: The length of the temporal window of integration (TWI) was shorter in adults (<200 ms) than older children (<300 ms in 9-11-year-olds) and younger children (<350 ms in the 5-8-year-olds). In addition, age-related changes were found in the latency and amplitude of the ERP components. CONCLUSIONS: The results demonstrate a general maturational development of the auditory evoked potentials and also a specific maturational process for temporal encoding of information in auditory cortex. SIGNIFICANCE: Rapid stimulus presentation rates can be successfully used in school-aged children to study neural mechanisms of auditory processes.     

Auditory cortex activation associated with octave illusion

Most right-handers perceive an octave illusion when they are presented with a 400 Hz tone to one ear and with a 800 Hz tone to the other ear simultaneously, and when the tones continuously reverse between the ears: instead of the correct sound sequence, the subjects typically report a high tone in the right ear alternating with a low tone in the left. To study the neural basis of the illusion, we recorded neuromagnetic responses to binaural 400 and 800 Hz tones in different combinations. In the auditory cortex of each hemisphere, the 100 ms response (N100m) was stronger to pairs where the 800 Hz tone was presented to the contralateral ear and the 400 Hz tone to the ipsilateral ear than vice versa. The sustained fields tended to behave in an opposite manner. We suggest that the perceived locations of the sounds in the octave illusion follow the N100m lateralization, and the percept is contributed by streaming by the ear.     

Organization of sequential sounds in auditory memory

A repeating five-tone pattern was presented at several stimulus rates (200, 400, 600, and 800 ms onset-to-onset) to determine at what temporal proximity the five-tone repeating unit would be represented in memory. The mismatch negativity component of event-related brain potentials was used to index how the sounds were organized in memory when participants had no task with the sounds. Only at the 200-ms onset-to-onset pace was the five-tone sequence unitized in memory. At presentation rates of 400 ms and above, the regularity (a different frequency tone occurred every fifth tone) was not detected and mismatch negativity was elicited by these tones in the sequence. The results show that temporal proximity plays a role in unitizing successive sounds in auditory memory. These results also suggest that global relationships between successive sounds are represented at the level of auditory cortices.     

Spectro-temporal analysis of complex tones: two cortical processes dependent on retention of sounds in the long auditory store.

OBJECTIVES: To examine whether two cortical processes concerned with spectro-temporal analysis of complex tones, a 'C-process' generating CN1 and CP2 potentials at cf. 100 and 180 ms after sudden change of pitch or timbre, and an 'M-process' generating MN1 and MP2 potentials of similar latency at the sudden cessation of repeated changes, are dependent on accumulation of a sound image in the long auditory store. METHODS: The durations of steady (440 Hz) and rapidly oscillating (440-494 Hz, 16 changes/s) pitch of a synthesized 'clarinet' tone were reciprocally varied between 0.5 and 4.5 s within a duty cycle of 5 s. Potentials were recorded at the beginning and end of the period of oscillation in 10 non-attending normal subjects. RESULTS: The CN1 at the beginning of pitch oscillation and the MN1 at the end were both strongly influenced by the duration of the immediately preceding stimulus pattern, mean amplitudes being 3-4 times larger after 4.5 s as compared with 0.5 s. CONCLUSIONS: The processes responsible for both CN1 and MN1 are influenced by the duration of the preceding sound pattern over a period comparable to that of the 'echoic memory' or long auditory store. The store therefore appears to occupy a key position in spectro-temporal sound analysis. The C-process is concerned with the spectral structure of complex sounds, and may therefore reflect the 'grouping' of frequency components underlying auditory stream segregation. The M-process (mismatch negativity) is concerned with the temporal sound structure, and may play an important role in the extraction of information from sequential sounds.     

Object representation in the human auditory system

One important principle of object processing is exclusive allocation. Any part of the sensory input, including the border between two objects, can only belong to one object at a time. We tested whether tones forming a spectro-temporal border between two sound patterns can belong to both patterns at the same time. Sequences were composed of low-, intermediate- and high-pitched tones. Tones were delivered with short onset-to-onset intervals causing the high and low tones to automatically form separate low and high sound streams. The intermediate-pitch tones could be perceived as part of either one or the other stream, but not both streams at the same time. Thus these tones formed a pitch 'border' between the two streams. The tones were presented in a fixed, cyclically repeating order. Linking the intermediate-pitch tones with the high or the low tones resulted in the perception of two different repeating tonal patterns. Participants were instructed to maintain perception of one of the two tone patterns throughout the stimulus sequences. Occasional changes violated either the selected or the alternative tone pattern, but not both at the same time. We found that only violations of the selected pattern elicited the mismatch negativity event-related potential, indicating that only this pattern was represented in the auditory system. This result suggests that individual sounds are processed as part of only one auditory pattern at a time. Thus tones forming a spectro-temporal border are exclusively assigned to one sound object at any given time, as are spatio-temporal borders in vision.     

Omission mismatch negativity of speech sounds reveals a functionally impaired temporal window of integration in schizophrenia

ObjectiveWe hypothesized that sensory memory associated with the temporal window of integration (TWI) would be impaired in patients with schizophrenia, an issue that had not been evaluated using omission mismatch negativity (MMN) of complex speech sounds. We aimed to assess the functional changes in auditory sensory memory associated with the TWI in patients with schizophrenia by investigating the effect of omission of complex speech stimuli on the MMN.MethodsIn total, 17 patients with schizophrenia and 15 control individuals participated in the study. The MMN in response to omission deviants of complex speech sounds was recorded, while the participants were instructed to ignore the series of speech sounds.ResultsThe MMN latency in patients with schizophrenia was significantly prolonged by deviant stimuli to omissions corresponding to the early and late parts of the temporal TWI. There were no significant group differences in the amplitude of the MMN to omissions at different time points across the TWI.ConclusionsOur results suggested that sensory tracing function in patients with schizophrenia is impaired in the early and the later half of the TWI.SignificanceWe showed that certain MMN abnormalities in patients with schizophrenia may be caused by an impaired TWI.     

Simultaneous storage of two complex temporal sound patterns in auditory sensory memory

We wished to determine whether multiple sound patterns can be simultaneously represented in the temporary auditory buffer (auditory sensory memory), when subjects have no task related to the sounds. To this end we used the mismatch negativity (MMN) event-related potential, an electric brain response elicited when a frequent sound is infrequently replaced by a different sound. The MMN response is based on the presence of the auditory sensory memory trace of the frequent sounds, which exists whether or not these sounds are in the focus of the subject's attention. Subjects watching a movie were presented with sound sequences consisting of two frequent sound patterns, each formed of four different tones and an infrequent pattern consisting of the first two tones of one of the frequent sound pattern and the last two tones of the other frequent sound pattern. The infrequent sound pattern elicited an MMN, indicating that multiple sound patterns are formed at an early, largely automatic stage of auditory processing.     

Abnormality of mismatch negativity in response to tone omission in dyslexic adults

Evidence of abnormalities in the perception of rapidly presented sounds in dyslexia has been interpreted as evidence of a prolonged time window within which sounds can influence the perception of temporally surrounding sounds. We recorded the magnetic mismatch negativity (MMNm) to infrequent tone omissions in a group of six dyslexic adults and six IQ and age-matched controls. An MMNm is only elicited in response to a complete stimulus omission when successive inputs fall within the temporal window of integration (stimulus onset asynchrony (SOA) approximately 160 ms). No MMNm responses were recorded in either experimental group when stimuli were presented at SOAs falling just outside the temporal window of integration (SOA = 175 ms). However, while presentation rates of 100 ms resulted in MMNm responses for all control participants, the same stimulus omissions elicited an MMNm response in only one of the six dyslexic participants. These results cannot support the hypothesis of a prolonged time window of integration, but rather indicate auditory grouping deficits in the dyslexic population.     

Attention effects on auditory scene analysis in children

Auditory scene analysis begins in infancy, making it possible for the baby to distinguish its mother's voice from other noises in the environment. Despite the importance of this process for human behavior, the question of how perceptual sound organization develops during childhood is not well understood. The current study investigated the role of attention for perceiving sound streams in a group of school-aged children and young adults. We behaviorally determined the frequency separation at which a set of sounds was detected as one integrated or two separated streams and compared these measures with passively and actively obtained electrophysiological indices (mismatch negativity (MMN) and P3b) of the same sounds. In adults, there was a high degree of concordance between passive and active electrophysiological indices of stream segregation that matched with perception. In contrast, there was a large disparity in children. Active electrophysiological indices of streaming were concordant with behavioral measures of perception, whereas passive indices were not. In addition, children required larger frequency separations to perceive two streams compared to adults. Our results suggest that differences in stream segregation between children and adults reflect an under-development of basic auditory processing mechanisms, and indicate a developmental role of attention for shaping physiological responses that optimize processes engaged during passive audition.