Phenomenology of the sound-induced flash illusion

Past studies, using pairings of auditory tones and visual flashes, which were static and coincident in space but variable in time, demonstrated errors in judging the temporal patterning of the visual flashes—the sound-induced flash illusion. These errors took one of the two forms: under-reporting (sound-induced fusion) or over-reporting (sound-induced fission) of the flash numbers. Our study had three objectives: to examine the robustness of both illusions and to consider the effects of stimulus set and response bias. To this end, we used an extended range of fixed spatial location flash–tone pairings, examined stimuli that were variable in space and time and measured confidence in judging flash numbers. Our results indicated that the sound-induced flash illusion is a robust percept, a finding underpinned by the confidence measures. Sound-induced fusion was found to be more robust than sound-induced fission and a most likely outcome when high numbers of flashes were incorporated within an incongruent flash–tone pairing. Conversely, sound-induced fission was the most likely outcome for the flash–tone pairing which contained two flashes. Fission was also shown to be strongly driven by stimuli confounds such as categorical boundary conditions (e.g. flash–tone pairings with ≤2 flashes) and compressed response options. These findings suggest whilst both fission and fusion are associated with ‘auditory driving’, the differences in the occurrence and strength of the two illusions not only reflect the separate neuronal mechanisms underlying audio and visual signal processing, but also the test conditions that have been used to investigate the sound-induced flash illusion.     

Top-down feature-based selection of matching features for audio-visual synchrony discrimination

Our previous findings suggest that audio-visual synchrony perception is based on the matching of salient temporal features selected from each sensory modality through bottom-up segregation or by top-down attention to a specific spatial position. This study examined whether top-down attention to a specific feature value is also effective in selection of cross-modal matching features. In the first experiment, the visual stimulus was a pulse train in which a flash randomly appeared with a probability of 6.25, 12.5 or 25% for every 6.25 ms. Four flash colors randomly appeared with equal probability, and one of them was selected as the target color on each trial. The paired auditory stimulus was a single-pitch pip sequence that had the same temporal structure as the target color flashes, presented in synchrony with the target flashes (synchronous stimulus) or with a 250-ms relative shift (asynchronous stimuli). The task of the participants was synchrony-asynchrony discrimination, with the target color being indicated to the participant by a probe (with-probe condition) or not (without probe). In another control condition, there was no correlation between color and auditory signals (color-shuffled). In the second experiment, the roles of visual and auditory stimuli were exchanged. The results show that the performance of synchrony-asynchrony discrimination was worst for the color/pitch-shuffled condition, but best under the with-probe condition where the observer knew beforehand which color/pitch should be matched with the signal of the other modality. This suggests that top-down, feature-based attention can aid in feature selection for audio-visual synchrony discrimination even when the bottom-up segmentation processes cannot uniquely determine salient features. The observed feature-based selection, however, is not as effective as position-based selection.     

Audio-visual integration in temporal perception.

In situations of audio-visual interaction, research has generally found that audition prevails over vision in temporal perception, while vision is dominant over audition for spatial perception. Modality appropriateness to a given task generally determines the direction of this inter-modality effect. However, we found a reverse effect in some situations where a change in the frequency of visual stimuli was associated with a perceived change in the frequency of auditory stimuli. In our experiment, 12 participants were asked to judge the change in the frequency of visual and auditory stimuli using a visual flicker and auditory flutter stimuli. In some conditions either the auditory or the visual information was ambiguous. In addition to confirming the expected finding that a change in the frequency of the auditory stimuli induced a perceived change in the frequency of the visual stimuli, we found a new phenomenon. When ambiguous auditory temporal cues were presented, the change in the frequency of the visual stimuli was associated with a perceived change in the frequency of the auditory stimuli. This suggests that cross-modal asymmetry effects are influenced by the reliability of visual and auditory information as well as modality appropriateness.     

Familiarity of objects affects susceptibility to the sound-induced flash illusion

Audition is accepted as more reliable (thus dominant) than vision when temporal discrimination is required by the task. However, it is not known whether the characteristics of the visual stimulus, for example its familiarity to the perceiver, affect auditory dominance. In this study we manipulated familiarity of the visual stimulus in a well-established multisensory phenomenon, i.e., the sound-induced flash illusion. This illusion occurs when, for example, one brief visual stimulus (e.g., a flash) is presented in close temporal proximity with two brief sounds; participants perceive two flashes instead of one. We found that when the visual stimuli (faces or buildings) were familiar, participants were less susceptible to the illusion than when they were unfamiliar. As the illusion has been ascribed to early cross-sensory interactions between vision and audition, the present work offers behavioural evidence that high level processing of objects' characteristics such as familiarity, affects early temporal multisensory integration. Possible mechanisms underlying the effect of familiarity are discussed.     

Early modulation of visual cortex by sound: an MEG study

Sound can alter visual perception. This has been recently demonstrated by a strong illusion in which a single flash is perceived as multiple flashes when accompanied by multiple brief sounds. While psychophysical findings on this sound-induced flash illusion indicate that the modulations of visual percept by sound occur at a perceptual processing level, it remains unclear at what level of perceptual processing these interactions occur and what mechanisms mediate them. Here we investigated these questions using MEG. We found modulation of activity in occipital and parietal scalp locations, when comparing illusion trials with visual-alone and auditory-alone trials. This modulation occurred as early as 35-65 ms from the onset of the visual stimulus. Activity was also modulated in the occipital and parietal areas as well as anterior areas at a later ( approximately 150 ms post-stimulus) onset. No significant interactions were observed in occipital and parietal areas in trials in which illusion was not perceived. These results indicate that the auditory alteration of visual perception as reflected by the illusion is associated with modulation of activity in visual cortex. The early onset of these modulations suggests that a feed-forward or lateral circuitry is at least partially involved in these interactions.     

Temporal ventriloquism: crossmodal interaction on the time dimension. 2. Evidence from sensorimotor synchronization.

In two experiments, we measured audio-visual crossmodal attraction on the time dimension, using a sensorimotor synchronization task. Synchronization performance made it possible to split up the total crossmodal attraction (demonstrated in earlier studies through inter-modal temporal order judgments) into its modality-specific components, the auditory bias of the visual event's perceived time of occurrence and the visual bias of the auditory event's perceived time of occurrence. Participants were asked to produce tapping movements in synchrony with a sequence of isochronously repeated pacing signals. In Experiment 1, pacing signals were light flashes, each preceded or followed, at one of several stimulus onset asynchronies (SOAs), by an auditory distracter that the participant was instructed to ignore. The timing of the tap was, in spite of that instruction, strongly biased toward the distracter. In Experiment 2, the converse task was used. The pacing signals were auditory and the to be ignored distracters, light flashes. The timing of the taps was biased significantly here also toward the distracter, but to a much lesser extent. Taken together, these results clearly demonstrate that audition plays a bigger role than vision in temporal ventriloquism and is probably generally superior to vision for processing the temporal dimension of events.     

Illusory double flashes can speed up responses like physical ones: evidence from the sound-induced flash illusion

When a single brief flash is accompanied by two auditory beeps, participants often report perceiving two flashes. The present experiment examined whether the perception of illusory redundant flashes can result in faster responses as compared to the perception of a single flash, because previous research has shown such a redundancy gain for physical stimuli. To this end, participants were asked to respond as rapidly as possible to the onset of any flash. Following their response, they additionally indicated whether they perceived a single flash or a double flash. Most importantly, we observed significant shorter reaction times in response to redundant flashes, irrespective of whether they were physically presented or illusorily perceived. Taken together, our results suggest that an illusory percept can affect simple reaction time in much the same manner as the corresponding physical stimulation.     

A dynamic fMRI study of illusory double-flash effect on human visual cortex

Functional MRI (fMRI) combined with the paired-stimuli paradigms (referred as dynamic fMRI) was used to study the “illusory double-flash” effect on brain activity in the human visual cortex. Three experiments were designed. The first two experiments aimed to examine the cross-modal neural interaction between the visual and auditory sensory systems caused by the illusory double-flash effect using combined auditory (beep sound) and visual (light flash) stimuli. The fMRI signal in the visual cortex was significantly increased in response to the illusory double flashes compared to the physical single flash when the inter-stimuli delay between the auditory and visual stimuli was 25 ms. This increase disappeared when the delay was prolonged to ~300 ms. These results reveal that the illusory double-flash effect can significantly affect the brain activity in the visual cortex, and the degree of this effect is dynamically sensitive to the inter-stimuli delay. The third experiment was to address the spatial differentiation of brain activation in the visual cortex in response to the illusory double-flash stimulation. It was found that the illusory double-flash effect in the human visual cortex is much stronger in the periphery than the fovea. This finding suggests that the periphery may be involved in high-level brain processing beyond the retinotopic visual perception. The behavioral measures conducted in this study indicate an excellent correlation between the fMRI results and behavioral performance. Finally, this work demonstrates a unique merit of fMRI for providing both temporal and spatial information regarding cross-modal neural interaction between different sensory systems.     

Sound-Induced Flash Illusion is Resistant to Feedback Training

A single flash accompanied by two auditory beeps tends to be perceived as two flashes (Shams et al. Nature 408:788, 2000, Cogn Brain Res 14:147–152, 2002). This phenomenon is known as ‘sound-induced flash illusion.’ Previous neuroimaging studies have shown that this illusion is correlated with modulation of activity in early visual cortical areas (Arden et al. Vision Res 43(23):2469–2478, 2003; Bhattacharya et al. NeuroReport 13:1727–1730, 2002; Shams et al. NeuroReport 12(17):3849–3852, 2001, Neurosci Lett 378(2):76–81, 2005; Watkins et al. Neuroimage 31:1247–1256, 2006, Neuroimage 37:572–578, 2007; Mishra et al. J Neurosci 27(15):4120–4131, 2007). We examined how robust the illusion is by testing whether the frequency of the illusion can be reduced by providing feedback. We found that the sound-induced flash illusion was resistant to feedback training, except when the amount of monetary reward was made dependent on accuracy in performance. However, even in the latter case the participants reported that they still perceived illusory two flashes even though they correctly reported single flash. Moreover, the feedback training effect seemed to disappear once the participants were no longer provided with feedback suggesting a short-lived refinement of discrimination between illusory and physical double flashes rather than vanishing of the illusory percept. These findings indicate that the effect of sound on the perceptual representation of visual stimuli is strong and robust to feedback training, and provide further evidence against decision factors accounting for the sound-induced flash illusion.     

Hearing the speed: visual motion biases the perception of auditory tempo

The coupling between sensory and motor processes has been established in various scenarios: for example, the perception of auditory rhythm entails an audiomotor representation of the sounds. Similarly, visual action patterns can also be represented via a visuomotor transformation. In this study, we tested the hypothesis that the visual motor information, such as embedded in a coherent motion flow, can interact with the perception of a motor-related aspect in auditory rhythm: the tempo. In the first two experiments, we employed an auditory tempo judgment task where participants listened to a standard auditory sequence while concurrently watching visual stimuli of different motion information, after which they judged the tempo of a comparison sequence related to the standard. In Experiment 1, we found that the same auditory tempo was perceived as faster when it was accompanied by accelerating visual motion than by non-motion luminance change. In Experiment 2, we compared the perceived auditory tempo among three visual motion conditions, increase in speed, decrease in speed, and no speed change, and found the corresponding bias in judgment of auditory tempo: faster than it was, slower than it was, and no bias. In Experiment 3, the perceptual bias induced by the change in motion speed was consistently reflected in the tempo reproduction task. Taken together, these results indicate that between a visual spatiotemporal and an auditory temporal stimulation, the embedded motor representations from each can interact across modalities, leading to a spatial-to-temporal bias. This suggests that the perceptual process in one modality can incorporate concurrent motor information from cross-modal sensory inputs to form a coherent experience.     

Responses of cat retinal ganglion cells to brief flashes of light

1. The responses of cat retinal ganglion cells to brief flashes of light have been illustrated and described with a view to providing material for comparison with psychophysical experiments in the scotopic (rod-dominated) range of performance.2. There is a minimum response duration of 50-70 msec no matter how brief the flash is made. This duration is reached with stimuli lasting 32 msec or shorter.3. Reducing the background illumination obviously increases the latency of responses to stimuli at 4 times threshold intensity (about 10 msec increment per log. unit decrement) but has no obvious effect on the minimum response duration.4. The relation between intensity and duration of a flash for threshold responses closely resembles that in human psychophysical experiments. The Bunsen-Roscoe law is applicable for flash durations up to 64 msec.5. If equal amounts of energy are delivered in the form of a pair of flashes of varying separation rather than by rectangular pulses, the shape of the response changes more abruptly with the temporal factor.6. Non-linear performance is apparent for stimuli as weak as 4 times threshold.7. A method is developed for quantitative analysis of individual responses. It is based upon cross-correlation of the train of impulses with a Gaussian smoothing function and represents local impulse frequency as a smooth function of time. The method also improves the signal-to-noise ratio of post-stimulus time-histograms of the sum of many responses.8. The measurement of variability of individual responses is the main result of the method; its magnitude indicates that it is a significant new factor limiting temporal resolution with suprathreshold stimuli.     

Modeling the hot flash experience in breast cancer survivors

OBJECTIVE: To evaluate relationships among different measures of hot flashes, perceived hot flash interference, and associated outcomes (positive affect, negative affect) while controlling potential covariates. DESIGN: Breast cancer survivors (N=236) provided demographic data, objective hot flash frequency data via sternal skin conductance monitoring, prospective diary-based hot flash frequency and severity data, and questionnaire data via the Hot Flash Related Daily Interference Scale and the Positive and Negative Affect Scale. RESULTS: Objective hot flash frequency and subjective hot flash severity emerged as separate factors in the structural equation model. Subjective hot flash frequency was associated with a high degree of unexplained variance (error) and seemed to be a potentially less accurate measure of either frequency or severity. Objective frequency was directly related to greater positive affect. In contrast, greater hot flash severity was (1) directly related to greater perceived hot flash interference and (2) indirectly related to more negative affect and lower positive affect through interference. CONCLUSIONS: Findings provide a theoretical basis for selecting among symptom measures and anticipating how interventions aimed at different hot flash measures might affect perceived hot flash interference or associated outcomes. Because objective hot flash frequency and subjective hot flash severity seemed to measure different dimensions, measuring both may provide a more comprehensive picture of women's symptom experiences.     

Sophisticated temporal pattern recognition in retinal ganglion cells

Pattern recognition is one of the most important tasks of the visual system, and uncovering the neural mechanisms underlying recognition phenomena has been a focus of researchers for decades. Surprisingly, at the earliest stages of vision, the retina is capable of highly sophisticated temporal pattern recognition. We stimulated the retina of tiger salamander (Ambystoma tigrinum) with periodic dark flash sequences and found that retinal ganglion cells had a wide variety of different responses to a periodic flash sequence with many firing when a flash was omitted. The timing of the omitted stimulus response (OSR) depended on the period, with individual cells tracking the stimulus period down to increments of 5 ms. When flashes occurred earlier than expected, cells updated their expectation of the next flash time by as much as 50 ms. When flashes occurred later than expected, cells fired an OSR and reset their temporal expectation to the average time interval between flashes. Using pharmacology to investigate the retinal circuitry involved, we found that inhibitory transmission from amacrine cells was not required, but on bipolar cells were required. The results suggest a mechanism in which the intrinsic resonance of on bipolars leads to the OSR in ganglion cells. We discuss the implications of retinal pattern recognition on the neural code of the retina and visual processing in general.     

Auditory grouping occurs prior to intersensory pairing: evidence from temporal ventriloquism

The authors examined how principles of auditory grouping relate to intersensory pairing. Two sounds that normally enhance sensitivity on a visual temporal order judgement task (i.e. temporal ventriloquism) were embedded in a sequence of flanker sounds which either had the same or different frequency (Exp. 1), rhythm (Exp. 2), or location (Exp. 3). In all experiments, we found that temporal ventriloquism only occurred when the two capture sounds differed from the flankers, demonstrating that grouping of the sounds in the auditory stream took priority over intersensory pairing. By combining principles of auditory grouping with intersensory pairing, we demonstrate that capture sounds were, counter-intuitively, more effective when their locations differed from that of the lights rather than when they came from the same position as the lights.     

Electrical cochlear stimulation in the deaf cat: comparisons between psychophysical and central auditory neuronal thresholds

Cochlear prostheses for electrical stimulation of the auditory nerve ("electrical hearing") can provide auditory capacity for profoundly deaf adults and children, including in many cases a restored ability to perceive speech without visual cues. A fundamental challenge in auditory neuroscience is to understand the neural and perceptual mechanisms that make rehabilitation of hearing possible in these deaf humans. We have developed a feline behavioral model that allows us to study behavioral and physiological variables in the same deaf animals. Cats deafened by injection of ototoxic antibiotics were implanted with either a monopolar round window electrode or a multichannel scala tympani electrode array. To evaluate the effects of perceptually significant electrical stimulation of the auditory nerve on the central auditory system, an animal was trained to avoid a mild electrocutaneous shock when biphasic current pulses (0.2 ms/phase) were delivered to its implanted cochlea. Psychophysical detection thresholds and electrical auditory brain stem response (EABR) thresholds were estimated in each cat. At the conclusion of behavioral testing, acute physiological experiments were conducted, and threshold responses were recorded for single neurons and multineuronal clusters in the central nucleus of the inferior colliculus (ICC) and the primary auditory cortex (A1). Behavioral and neurophysiological thresholds were evaluated with reference to cochlear histopathology in the same deaf cats. The results of the present study include: 1) in the cats implanted with a scala tympani electrode array, the lowest ICC and A1 neural thresholds were virtually identical to the behavioral thresholds for intracochlear bipolar stimulation; 2) behavioral thresholds were lower than ICC and A1 neural thresholds in each of the cats implanted with a monopolar round window electrode; 3) EABR thresholds were higher than behavioral thresholds in all of the cats (mean difference = 6.5 dB); and 4) the cumulative number of action potentials for a sample of ICC neurons increased monotonically as a function of the amplitude and the number of stimulating biphasic pulses. This physiological result suggests that the output from the ICC may be integrated spatially across neurons and temporally integrated across pulses when the auditory nerve array is stimulated with a train of biphasic current pulses. Because behavioral thresholds were lower and reaction times were faster at a pulse rate of 30 pps compared with a pulse rate of 2 pps, spatial-temporal integration in the central auditory system was presumably reflected in psychophysical performance.     

Audiovisual time perception is spatially specific

Our sensory systems face a daily barrage of auditory and visual signals whose arrival times form a wide range of audiovisual asynchronies. These temporal relationships constitute an important metric for the nervous system when surmising which signals originate from common external events. Internal consistency is known to be aided by sensory adaptation: repeated exposure to consistent asynchrony brings perceived arrival times closer to simultaneity. However, given the diverse nature of our audiovisual environment, functionally useful adaptation would need to be constrained to signals that were generated together. In the current study, we investigate the role of two potential constraining factors: spatial and contextual correspondence. By employing an experimental design that allows independent control of both factors, we show that observers are able to simultaneously adapt to two opposing temporal relationships, provided they are segregated in space. No such recalibration was observed when spatial segregation was replaced by contextual stimulus features (in this case, pitch and spatial frequency). These effects provide support for dedicated asynchrony mechanisms that interact with spatially selective mechanisms early in visual and auditory sensory pathways.     

Synesthetic congruency modulates the temporal ventriloquism effect

People sometimes find it easier to judge the temporal order in which two visual stimuli have been presented if one tone is presented before the first visual stimulus and a second tone is presented after the second visual stimulus. This enhancement of people's visual temporal sensitivity has been attributed to the temporal ventriloquism of the visual stimuli toward the temporally proximate sounds, resulting in an expansion of the perceived interval between the two visual events. In the present study, we demonstrate that the synesthetic congruency between the auditory and visual stimuli (in particular, between the relative pitch of the sounds and the relative size of the visual stimuli) can modulate the magnitude of this multisensory integration effect: The auditory capture of vision is larger for pairs of auditory and visual stimuli that are synesthetically congruent than for pairs of stimuli that are synesthetically incongruent, as reflected by participants' increased sensitivity in discriminating the temporal order of the visual stimuli. These results provide the first evidence that multisensory temporal integration can be affected by the synesthetic congruency between the auditory and visual stimuli that happen to be presented.     

Characteristics of the pupillary light reflex in the macaque monkey: metrics

To investigate whether the simian light reflex is a reasonable model for the human light reflex, we elicited pupillary responses in three behaving rhesus macaques. We measured the change in pupillary area in response to brief (100 ms), intermediate (1 s), and long (3-5 s) light flashes delivered by light-emitting diodes while the monkey fixated a stationary target. Individual responses in the same monkey to either 100-ms or 1-s stimuli of the same light intensity were quite variable. Nevertheless, in response to the 100-ms stimulus, average pupillary constriction and peak constriction velocity increased and latency decreased linearly with the log of stimulus luminance. The minimum average constriction latency across monkeys for the brightest flash was 136 ms. A linear decrease of constriction latency with stimulus luminance also occurs in humans, but their latencies are approximately 70 ms longer. In addition, peak constriction velocity was highly correlated with the decrease in pupillary area. Dilation metrics were not as well related to stimulus luminance as were constriction metrics. The latency from flash offset to the onset of dilation was relatively constant, averaging approximately 480 ms. Peak dilation velocity was also correlated, but less well, with the increase in pupillary area. Constriction generally was greater and of longer duration for 1-s light pulses than for 100-ms pulses of equal luminance. The initial time courses of the responses to the two stimuli of different durations were identical until approximately 150 ms after response onset. Human pupillary responses for long and short flashes also have identical initial time courses. For very long (3-5 s) and very bright constant-luminance stimuli, the simian pupil underwent oscillations at frequencies of 0.9-1.6 Hz. Similar oscillations, called hippus, occur in the human pupillary light reflex. Like humans, the monkeys also exhibited consensual and binocular pupillary responses. Except for response latency, the pupillary responses in the two primate species are otherwise quite similar. Therefore any knowledge we gain about the neuronal substrate of the simian light reflex can be expected to have considerable relevance when extrapolated to humans.     

Complex temporal response patterns with a simple retinal circuit

The retina can respond to a wide array of features in the visual input. It was recently reported that the retina can even recognize complicated temporal input patterns and signal violations in the patterns. When a sequence of flashes was presented, ganglion cells exhibited a variety of firing profiles and many cells showed an "omitted stimulus response" (OSR), in which they fired strongly if a flash in the sequence was omitted. We examined the synaptic origins of the OSR by recording excitatory synaptic currents from ganglion cells in the salamander retina in response to periodic flash sequences. Consistent with previous spike recordings, ganglion cells exhibited an OSR in their current response and the OSR shifted in time with a change in flash frequency such that it could predict when the next flash should have occurred. Although the behavior may seem sophisticated, we show that a simple linear-nonlinear model with a spike threshold can account for the OSR in on ganglion cells and that the variety of complex firing profiles seen in other ganglion cells can be explained by adding contributions from the off pathway. We discuss the physiological and simulation results and their implications for understanding retinal mechanisms of visual information processing.     

Responses of inferior colliculus neurons to amplitude-modulated intracochlear electrical pulses in deaf cats

Current cochlear prostheses use amplitude-modulated pulse trains to encode acoustic signals. In this study we examined the responses of inferior colliculus (IC) neurons to sinusoidal amplitude-modulated pulses and compared the maximum unmodulated pulse rate (Fmax) to which they responded with the maximum modulation frequency (maxFm) that they followed. Consistent with previous results, responses to unmodulated pulses were all low-pass functions of pulse rate. Mean Fmax to unmodulated pulses was 104 pulses per second (pps) and modal Fmax was 60 pps. Above Fmax IC neurons ceased responding except for an onset burst at the beginning of the stimulus. However, IC neurons responded to much higher pulse rates when these pulses were amplitude modulated; 74% were relatively insensitive to carrier rate and responded to all modulated carriers including those exceeding 600 pps. In contrast, the responses of these neurons (70%) were low-pass functions of modulation frequency, and the remaining (30%) had band-pass functions with a maxFm of 42 and 34 Hz, respectively. Thus temporal resolution of IC neurons for modulated frequencies is significantly lower than that for unmodulated pulses. These two measures of temporal resolution (Fmax and maxFm) were uncorrelated (r(2) = 0.101). Several parameters influenced the amplitude and temporal structure of modulation responses including modulation depth, overall intensity and modulation-to-carrier rate ratio. We observed distortions in unit responses to amplitude-modulated signals when this ratio was 1/4 to 1/6. Since most current cochlear implant speech processors permit ratios that are significantly greater than this, severe distortion and signal degradation may occur frequently in these devices.