Audiovisual temporal adaptation of speech: temporal order versus simultaneity judgments

The temporal perception of simple auditory and visual stimuli can be modulated by exposure to asynchronous audiovisual speech. For instance, research using the temporal order judgment (TOJ) task has shown that exposure to temporally misaligned audiovisual speech signals can induce temporal adaptation that will influence the TOJs of other (simpler) audiovisual events (Navarra et al. (2005) Cognit Brain Res 25:499–507). Given that TOJ and simultaneity judgment (SJ) tasks appear to reflect different underlying mechanisms, we investigated whether adaptation to asynchronous speech inputs would also influence SJ task performance. Participants judged whether a light flash and a noise burst, presented at varying stimulus onset asynchronies, were simultaneous or not, or else they discriminated which of the two sensory events appeared to have occurred first. While performing these tasks, participants monitored a continuous speech stream for target words that were either presented in synchrony, or with the audio channel lagging 300 ms behind the video channel. We found that the sensitivity of participant’s TOJ and SJ responses was reduced when the background speech stream was desynchronized. A significant modulation of the point of subjective simultaneity (PSS) was also observed in the SJ task but, interestingly, not in the TOJ task, thus supporting previous claims that TOJ and SJ tasks may tap somewhat different aspects of temporal perception.

Methodological problems undermine tests of the ideo-motor conjecture

Recent behavioural research has investigated whether viewing someone perform an action results in activation of that action by the observer. Postulated empirical support for this ‘ideo-motor (IM) conjecture’ typically rests upon two types of experimental paradigm (reaction time and movement tracking tasks). These paradigms purport to show movement facilitation when compatible movements are observed and vice versa, but only for biological stimuli. Unfortunately, these paradigms often contain confounding (and unavoidable) generic stimulus–response compatibility effects that are not restricted to observed human movement. The current study demonstrates in three experiments that equivalent compatibility effects can be produced by non-biological stimuli. These results suggest that existing empirical paradigms may not, and perhaps cannot, support the IM-conjecture.

Attention modulates the specificity of automatic imitation to human actors

The perception of actions performed by others activates one’s own motor system. Recent studies disagree as to whether this effect is specific to actions performed by other humans, an issue complicated by differences in perceptual salience between human and non-human stimuli. We addressed this issue by examining the automatic imitation of actions stimulated by viewing a virtual, computer-generated, hand. This stimulus was held constant across conditions, but participants’ attention to the virtualness of the hand was manipulated by informing some participants during instructions that they would see a “computer-generated model of a hand,” while making no mention of this to others. In spite of this attentional manipulation, participants in both conditions were generally aware of the virtualness of the hand. Nevertheless, automatic imitation of the virtual hand was significantly reduced––but not eliminated––when participants were told they would see a virtual hand. These results demonstrate that attention modulates the “human bias” of automatic imitation to non-human actors.

Semantic congruency and the Colavita visual dominance effect

Participants presented with auditory, visual, or bimodal audiovisual stimuli in a speeded discrimination task, fail to respond to the auditory component of bimodal targets significantly more often than to the visual component, a phenomenon known as the Colavita visual dominance effect. Given that spatial and temporal factors have recently been shown to modulate the Colavita effect, the aim of the present study, was to investigate whether semantic congruency also modulates the effect. In the three experiments reported here, participants were presented with a version of the Colavita task in which the stimulus congruency between the auditory and visual components of the bimodal targets was manipulated. That is, the auditory and visual stimuli could refer to the same or different object (in Experiments 1 and 2) or audiovisual speech event (Experiment 3). Surprisingly, semantic/stimulus congruency had no effect on the magnitude of the Colavita effect in any of the experiments, although it exerted a significant effect on certain other aspects of participants’ performance. This finding contrasts with the results of other recent studies showing that semantic/stimulus congruency can affect certain multisensory interactions.

Involuntary cueing effects during smooth pursuit: facilitation and inhibition of return in oculocentric coordinates

Peripheral cues induce facilitation with short cue-target intervals and inhibition of return (IOR) with long cue-target intervals. Modulations of facilitation and IOR by continuous displacements of the eye or the cued stimuli are poorly understood. Previously, the retinal coordinates of the cued location were changed by saccadic or smooth pursuit eye movements during the cue-target interval. In contrast, we probed the relevant coordinates for facilitation and IOR by orthogonally varying object motion (stationary, moving) and eye movement (fixation, smooth pursuit). In the pursuit conditions, cue and target were presented during the ongoing eye movement and observers made a saccade to the target. Importantly, we found facilitation and IOR of similar size during smooth pursuit and fixation. The results suggest that involuntary orienting is possible even when attention has to be allocated to the moving target during smooth pursuit. Comparison of conditions with stabilized and moving objects suggest an oculocentric basis for facilitation as well as inhibition. Facilitation and IOR were reduced with objects that moved on the retina both with smooth pursuit and eye fixation.

The hand’s automatic pilot can update visual information while the eye is in motion

When participants reach for a target, their hand can adjust to a change in target position that occurs while their eyes are in motion (the hand’s automatic pilot) even though they are not aware of the target’s displacement (saccadic suppression of perceptual experience). However, previous studies of this effect have displayed the target without interruption, such that the new target position remains visible during the fixation that follows the saccade. Here we test whether a change in target position that begins and ends during the saccade can be used to update aiming movements. We also ask whether such information can be acquired from two targets at a time. The results showed that participants responded to single and double target jumps even when these targets were extinguished prior to saccade termination. The results imply that the hand’s automatic pilot is updated with new visual information even when the eye is in motion.

Misperceiving the speed-accuracy tradeoff: imagined movements and perceptual decisions

Research has suggested that prospective motor decisions are consistent with actual motor action. In a study that we recently published (Young et al. in Exp Brain Res 185:681–688, 2008), however, participants demonstrated a preference for closer targets that was inconsistent with the predictions of Fitts’s law. With a pair of experiments, the present paper investigates the underlying basis of this non-optimal behaviour. Participants showed a similar deviation from Fitts’s law when imagining movements—believing that movement duration increased with distance within the same index of difficulty. Participants did not behave similarly, however, in a perceptual version of the decision task. These results suggest that imagined movements and motor decisions are linked, as well as demonstrating one situation in which both show a similar deviation from the patterns of actual movement duration.

Evidence for an attentional component in saccadic inhibition of return

After presentation of a peripheral cue, facilitation at the cued location is followed by inhibition of return (IOR). It has been recently proposed that IOR may originate at different processing stages for manual and ocular responses, with manual IOR resulting from inhibited attentional orienting, and ocular IOR resulting form inhibited motor preparation. Contrary to this interpretation, we found an effect of target contrast on saccadic IOR. The effect of contrast decreased with increasing reaction times (RTs) for saccades, but not for manual key-press responses. This may have masked the effect of contrast on IOR with saccades in previous studies (Hunt and Kingstone in J Exp Psychol Hum Percept Perform 29:1068–1074, 2003) because only mean RTs were considered. We also found that background luminance strongly influenced the effects of gap and target contrast on IOR.

When vision ‘extinguishes’ touch in neurologically-normal people: extending the Colavita visual dominance effect

Research has shown that people fail to report the presence of the auditory component of suprathreshold audiovisual targets significantly more often than they fail to detect the visual component in speeded response tasks. Here, we investigated whether this phenomenon, known as the “Colavita effect”, also affects people’s perception of visuotactile stimuli as well. In Experiments 1 and 2, participants made speeded detection/discrimination responses to unimodal visual, unimodal tactile, and bimodal (visual and tactile) stimuli. A significant Colavita visual dominance effect was observed (i.e., participants failed to respond to touch far more often than they failed to respond to vision on the bimodal trials). This dominance of vision over touch was significantly larger when the stimuli were presented from the same position than when they were presented from different positions (Experiment 3), and still occurred even when the subjective intensities of the visual and tactile stimuli had been matched (Experiment 4), thus ruling out a simple intensity-based account of the results. These results suggest that the Colavita visual dominance effect (over touch) may result from a competition between the neural representations of the two stimuli for access to consciousness and/or the recruitment of attentional resources.

Role of the lateral prefrontal cortex in visual object-based selective attention

We demonstrate that attention to object representations is vitally dependent on the prefrontal cortex. Object-based selective attention was compared in neurologic patients with unilateral damage to either the dorsolateral prefrontal cortex (DLPFC) or the parietal cortex and in healthy controls. Our task required a top–down attentional modulation of object representations in which spatial location played no role. All groups could invoke top–down object-based selection, but the DLPFC patients showed a selective deficit when target stimuli were in the hemifield contralateral to the lesioned hemisphere. Our findings indicate that in the healthy brain, anterior cortical mechanisms are crucial for attending to object-centered representations, whereas posterior cortical mechanisms are necessary for attending to objects at locations in the visual scene.

The impact of real and illusory target perturbations on manual aiming

This experiment was designed to determine if real and illusory shifts in target position at movement initiation affect the same online corrective processes. Adult participants completed rapid goal-directed movements toward the vertex of a target “T” located at the midline, 25 cm distal to a small home position. At movement initiation, the target either stayed the same, shifted its real position, its illusory position or both. The real perturbation involved a 2.5 mm shift either toward or away from the body. For the illusory perturbation, the horizontal portion of the “T” changed to inward or outward Müller–Lyer wings. Both the real and the illusory perturbation affected movement outcome. The two manipulations began to have their impact at peak velocity. Because both perturbations affected mid to late trajectory control and because their effects were not independent, we concluded that real and illusory target shifts impact late visual motor control associated with a comparison between the position of the limb and the perceived position of the target.

---

Crossmodal interaction in saccadic reaction time: separating multisensory from warning effects in the time window of integration model

In a focused attention task saccadic reaction time (SRT) to a visual target stimulus (LED) was measured with an auditory (white noise burst) or tactile (vibration applied to palm) non-target presented in ipsi- or contralateral position to the target. Crossmodal facilitation of SRT was observed under all configurations and stimulus onset asynchrony (SOA) values ranging from  −500 (non-target prior to target) to 0 ms, but the effect was larger for ipsi- than for contralateral presentation within an SOA range from  −200 ms to 0. The time-window-of-integration (TWIN) model (Colonius and Diederich in J Cogn Neurosci 16:1000, 2004) is extended here to separate the effect of a spatially unspecific warning effect of the non-target from a spatially specific and genuine multisensory integration effect.

Age-related differences in corrected and inhibited pointing movements

It has been widely reported that aging is accompanied by a decline in motor skill performance and in particular, it has been shown that older subjects take longer to adapt their ongoing reach in response to a target location shift. In the present experiment, we investigated the influence of aging on the ability to perform trajectory corrections in response to a target jump, but also assessed inhibition by asking a younger and an older group of participants to either adapt or stop their ongoing movement in response to a target location change. Results showed that although older subjects took longer to initiate, execute, correct and inhibit an ongoing reach, they performed both tasks with the same level of accuracy as the younger sample. Moreover, the slowing was also observed when older subjects were asked to point to stationary targets. Our findings thus indicate that aging does not specifically influence the ability to perform or inhibit fast online corrections to target location changes, but rather produces a general slowing and increased variability of movement planning, initiation and execution to both perturbed and stationary targets. For the first time, we demonstrate that aging is not accompanied by a decrease in the inhibition of motor control.

Integration of auditory and visual information in the recognition of realistic objects

Recognizing a natural object requires one to pool information from various sensory modalities, and to ignore information from competing objects. That the same semantic knowledge can be accessed through different modalities makes it possible to explore the retrieval of supramodal object concepts. Here, object-recognition processes were investigated by manipulating the relationships between sensory modalities, specifically, semantic content, and spatial alignment between auditory and visual information. Experiments were run under realistic virtual environment. Participants were asked to react as fast as possible to a target object presented in the visual and/or the auditory modality and to inhibit a distractor object (go/no-go task). Spatial alignment had no effect on object-recognition time. The only spatial effect observed was a stimulus–response compatibility between the auditory stimulus and the hand position. Reaction times were significantly shorter for semantically congruent bimodal stimuli than would be predicted by independent processing of information about the auditory and visual targets. Interestingly, this bimodal facilitation effect was twice as large as found in previous studies that also used information-rich stimuli. An interference effect was observed (i.e. longer reaction times to semantically incongruent stimuli than to the corresponding unimodal stimulus) only when the distractor was auditory. When the distractor was visual, the semantic incongruence did not interfere with object recognition. Our results show that immersive displays with large visual stimuli may provide large multimodal integration effects, and reveal a possible asymmetry in the attentional filtering of irrelevant auditory and visual information.

Crossmodal exogenous orienting improves the accuracy of temporal order judgments

Although many studies have demonstrated that crossmodal exogenous orienting can lead to a facilitation of reaction times, the issue of whether exogenous spatial orienting also affects the accuracy of perceptual judgments has proved to be much more controversial. Here, we examined whether or not exogenous spatial attentional orienting would affect sensitivity in a temporal discrimination task. Participants judged which of the two target letters, presented on either the same or opposite sides, had been presented first. A spatially non-predictive tone was presented 200 ms prior to the onset of the first visual stimulus. In two experiments, we observed improved performance (i.e., a decrease in the just noticeable difference) when the target letters were presented on opposite sides and the auditory cue was presented on the side of the first visual stimulus, even when central fixation was monitored ("Experiment 2"). A shift in the point of subjective simultaneity was also observed in both experiments, indicating ‘prior entry’ for cued as compared to uncued first target trials. No such JND or PSS effects were observed when the auditory tone was presented after the second visual stimulus ("Experiment 3"), thus confirming the attentional nature of the effects observed. These findings clearly show that the crossmodal exogenous orienting of spatial attention can affect the accuracy of temporal judgments.

Accommodation and Size-Constancy of Virtual Objects

Accommodation has been suspected as a contributor to size illusions in virtual environments (VE) due to the lack of appropriate accommodative stimuli in a VE for the objects displayed. Previous experiments examining size-constancy in VE have shown that monocular cues to depth that accompany the object are a major contributor to correct size perception. When these accompanying cues are removed perceived size varied with the object’s distance from the subject, i.e., visual angle. If accommodation were the dominant mechanism contributing to a visual angle response [due to its action to keep physical objects clear] in this condition, an open-loop accommodation viewing condition might restore size-constancy to this condition. Pinhole apertures were used to open-loop accommodation and examine if size-constancy might be restored when few accompanying monocular cues to depth were present. Visual angle performance when viewing a low cue environment was found with and without the use of the pinhole apertures. Thus, these results signify that accommodation does not play a dominate role in the loss of size-constancy in sparse visual environments often used in VE. These results suggest that size-constancy is driven by the inclusion of the remaining monocular cues to depth in VE as it is in the physical world.

The effect of sound intensity on the audiotactile crossmodal dynamic capture effect

We investigated the effect of varying sound intensity on the audiotactile crossmodal dynamic capture effect. Participants had to discriminate the direction of a target stream (tactile, Experiment 1; auditory, Experiment 2) while trying to ignore the direction of a distractor stream presented in a different modality (auditory, Experiment 1; tactile, Experiment 2). The distractor streams could either be spatiotemporally congruent or incongruent with respect to the target stream. In half of the trials, the participants were presented with auditory stimuli at 75 dB(A) while in the other half of the trials they were presented with auditory stimuli at 82 dB(A). Participants’ performance on both tasks was significantly affected by the intensity of the sounds. Namely, the crossmodal capture of tactile motion by audition was stronger with the more intense (vs. less intense) auditory distractors (Experiment 1), whereas the capture effect exerted by the tactile distractors was stronger for less intense (than for more intense) auditory targets (Experiment 2). The crossmodal dynamic capture was larger in Experiment 1 than in Experiment 2, with a stronger congruency effect when the target streams were presented in the tactile (vs. auditory) modality. Two explanations are put forward to account for these results: an attentional biasing toward the more intense auditory stimuli, and a modulation induced by the relative perceptual weight of, respectively, the auditory and the tactile signals.

Typical object velocity influences motion extrapolation

Previous work indicates that extrapolation of object motion during occlusion is affected by the velocity of the immediately preceding trial. Here we ask whether longer-term velocity representations can also influence motion extrapolation. Red, blue or green targets disappeared behind an occluder. Participants pressed a button when they thought the target had reached the other side. Red targets were slower (10–20 deg/s), blue targets moved at medium velocities (14–26 deg/s) and green targets were faster (20–30 deg/s). We compared responses on a subset of red and green trials which always travelled at 20 deg/s. Although trial velocities were identical, participants responded as if the green targets moved faster (M = 22.64 deg/s) then the red targets (M = 19.72 deg/s). This indicates that motion extrapolation is affected by longer-term information about the typical velocity of different categories of stimuli.

Perception of affordances for standing on an inclined surface depends on height of center of mass

We investigated whether perception of affordances for standing on an inclined surface depended on the height of center of mass of the perceiver-actor. Participants adjusted the angle of inclination of a surface until they felt that it was just barely possible for them to stand on that surface. They performed this task while wearing a backpack apparatus to which masses were attached in one of three configurations—high-mass, low-mass, and no-mass. Moreover, participants performed this task by viewing the inclined surface or by probing it with a hand-held rod (while blindfolded). Perception of affordances for standing on the inclined surface reflected the changes in center of mass brought on by the weighted backpack apparatus (the perceptual boundary occurred at a smaller angle of inclination in the high-mass condition than in the low-mass condition and in the no-mass condition). Moreover, perception of this affordance reflected such changes both when the surface was viewed and when the surface was probed with a hand-held rod (while blindfolded). The results highlight that perception of affordances is dynamic and task-dependent and suggest that the stimulation patterns that support perception of affordances are invariant and modality-independent.

The selective effect of the image of a hand on visuotactile interactions as assessed by performance on the crossmodal congruency task

Seeing one’s own body (either directly or indirectly) can influence visuotactile crossmodal interactions. Previously, it has been shown that even viewing a simple line drawing of a hand can also modulate such crossmodal interactions, as if viewing the picture of a hand somehow primes the representation of one’s own hand. However, factors other than the sight of a symbolic picture of a hand may have modulated the crossmodal interactions reported in previous research. In the present study, we examined the crossmodal modulatory effects of viewing five different visual images (photograph of a hand, line drawing of a hand, line drawing of a car, an U-shape, and an ellipse) on tactile performance. Participants made speeded discrimination responses regarding the location of brief vibrotactile targets presented to either the tip or base of their left index finger, while trying to ignore visual distractors presented to either the left or right of central fixation. We compared the visuotactile congruency effects elicited when the five different visual images were presented superimposed over the visual distractors. Participants’ tactile discrimination performance was modulated to a significantly greater extent by viewing the photograph of a hand than when viewing the outline drawing of a hand. No such crossmodal congruency effects were reported in any of the other conditions. These results therefore suggest that visuotactile interactions are specifically modulated by the image of the hand rather than just by any simple orientation cues that may be provided by the image of a hand.