Binding of movement, sound and touch: multimodal coordination dynamics

Very little is known about the coordination of movement in combination with stimuli such as sound and touch. The present research investigates the hypothesis that both the type of action (e.g., a flexion or extension movement) and the sensory modality (e.g., auditory or tactile) determine the stability of multimodal coordination. We performed a parametric study in which the ability to synchronize movement, touch and sound was explored over a broad range of stimulus frequencies or rates. As expected, synchronization of finger movement with external auditory and tactile stimuli was successfully established and maintained across all frequencies. In the key experimental conditions, participants were instructed to synchronize peak flexion of the index finger with touch and peak extension with sound (and vice-versa). In this situation, tactile and auditory stimuli were delivered counter-phase to each other. Two key effects were observed. First, switching between multimodal coordination patterns occurred, with transitions selecting one multimodal pattern (flexion with sound and extension with touch) more often than its partner. This finding indicates that the stability of multimodal coordination is influenced by both the type of action and the stimulus modality. Second, at higher rates, transitions from coherent to incoherent phase relations between touch, movement and sound occurred, attesting to the breakdown of multimodal coordination. Because timing errors in multimodal coordination were systematically altered when compared to unimodal control conditions we are led to consider the role played by time delays in multimodal coordination dynamics.