Two cortical areas mediate multisensory integration in superior colliculus neurons

The majority of multisensory neurons in the cat superior colliculus (SC) are able to synthesize cross-modal cues (e.g., visual and auditory) and thereby produce responses greater than those elicited by the most effective single modality stimulus and, sometimes, greater than those predicted by the arithmetic sum of their modality-specific responses. The present study examined the role of corticotectal inputs from two cortical areas, the anterior ectosylvian sulcus (AES) and the rostral aspect of the lateral suprasylvian sulcus (rLS), in producing these response enhancements. This was accomplished by evaluating the multisensory properties of individual SC neurons during reversible deactivation of these cortices individually and in combination using cryogenic deactivation techniques. Cortical deactivation eliminated the characteristic multisensory response enhancement of nearly all SC neurons but generally had little or no effect on a neuron's modality-specific responses. Thus, the responses of SC neurons to combinations of cross-modal stimuli were now no different from those evoked by one or the other of these stimuli individually. Of the two cortical areas, AES had a much greater impact on SC multisensory integrative processes, with nearly half the SC neurons sampled dependent on it alone. In contrast, only a small number of SC neurons depended solely on rLS. However, most SC neurons exhibited dual dependencies, and their multisensory enhancement was mediated by either synergistic or redundant influences from AES and rLS. Corticotectal synergy was evident when deactivating either cortical area compromised the multisensory enhancement of an SC neuron, whereas corticotectal redundancy was evident when deactivation of both cortical areas was required to produce this effect. The results suggest that, although multisensory SC neurons can be created as a consequence of a variety of converging tectopetal afferents that are derived from a host of subcortical and cortical structures, the ability to synthesize cross-modal inputs, and thereby produce an enhanced multisensory response, requires functional inputs from the AES, the rLS, or both.     

Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli

Many neurons in the superior colliculus (SC) integrate sensory information from multiple modalities, giving rise to significant response enhancements. Although enhanced multisensory responses have been shown to depend on the spatial and temporal relationships of the stimuli as well as on their relative effectiveness, these factors alone do not appear sufficient to account for the substantial heterogeneity in the magnitude of the multisensory products that have been observed. Toward this end, the present experiments have revealed that there are substantial differences in the operations used by different multisensory SC neurons to integrate their cross-modal inputs, suggesting that intrinsic differences in these neurons may also play an important deterministic role in multisensory integration. In addition, the integrative operation employed by a given neuron was found to be well correlated with the neuron's dynamic range. In total, four categories of SC neurons were identified based on how their multisensory responses changed relative to the predicted addition of the two unisensory inputs as stimulus effectiveness was altered. Despite the presence of these categories, a general rule was that the most robust multisensory enhancements were seen with combinations of the least effective unisensory stimuli. Together, these results provide a better quantitative picture of the integrative operations performed by multisensory SC neurons and suggest mechanistic differences in the way in which these neurons synthesize cross-modal information.     

Neonatal cortical ablation disrupts multisensory development in superior colliculus

The ability of cat superior colliculus (SC) neurons to synthesize information from different senses depends on influences from two areas of the cortex: the anterior ectosylvian sulcus (AES) and the rostral lateral suprasylvian sulcus (rLS). Reversibly deactivating the inputs to the SC from either of these areas in normal adults severely compromises this ability and the SC-mediated behaviors that depend on it. In this study, we found that removal of these areas in neonatal animals precluded the normal development of multisensory SC processes. At maturity there was a substantial decrease in the incidence of multisensory neurons, and those multisensory neurons that did develop were highly abnormal. Their cross-modal receptive field register was severely compromised, as was their ability to integrate cross-modal stimuli. Apparently, despite the impressive plasticity of the neonatal brain, it cannot compensate for the early loss of these cortices. Surprisingly, however, neonatal removal of either AES or rLS had comparatively minor consequences on these properties. At maturity multisensory SC neurons were quite common: they developed the characteristic spatial register among their unisensory receptive fields and exhibited normal adult-like multisensory integration. These observations suggest that during early ontogeny, when the multisensory properties of SC neurons are being crafted, AES and rLS may have the ability to compensate for the loss of one another's cortico-collicular influences so that normal multisensory processes can develop in the SC.     

Cortex controls multisensory depression in superior colliculus

Multisensory depression is a fundamental index of multisensory integration in superior colliculus (SC) neurons. It is initiated when one sensory stimulus (auditory) located outside its modality-specific receptive field degrades or eliminates the neuron's responses to another sensory stimulus (visual) presented within its modality-specific receptive field. The present experiments demonstrate that the capacity of SC neurons to engage in multisensory depression is strongly dependent on influences from two cortical areas (the anterior ectosylvian and rostral lateral suprasylvian sulci). When these cortices are deactivated, the ability of SC neurons to synthesize visual-auditory inputs in this way is compromised; multisensory responses are disinhibited, becoming more vigorous and in some cases indistinguishable from responses to the visual stimulus alone. Although obtaining a more robust multisensory SC response when cortex is nonfunctional than when it is functional may seem paradoxical, these data may help explain previous observations that the loss of these cortical influences permits visual orientation behavior in the presence of a normally disruptive auditory stimulus.     

Alterations to multisensory and unisensory integration by stimulus competition

In environments containing sensory events at competing locations, selecting a target for orienting requires prioritization of stimulus values. Although the superior colliculus (SC) is causally linked to the stimulus selection process, the manner in which SC multisensory integration operates in a competitive stimulus environment is unknown. Here we examined how the activity of visual-auditory SC neurons is affected by placement of a competing target in the opposite hemifield, a stimulus configuration that would, in principle, promote interhemispheric competition for access to downstream motor circuitry. Competitive interactions between the targets were evident in how they altered unisensory and multisensory responses of individual neurons. Responses elicited by a cross-modal stimulus (multisensory responses) proved to be substantially more resistant to competitor-induced depression than were unisensory responses (evoked by the component modality-specific stimuli). Similarly, when a cross-modal stimulus served as the competitor, it exerted considerably more depression than did its individual component stimuli, in some cases producing more depression than predicted by their linear sum. These findings suggest that multisensory integration can help resolve competition among multiple targets by enhancing orientation to the location of cross-modal events while simultaneously suppressing orientation to events at alternate locations.     

Time course of auditory masker effects: Tapping the locus of audiovisual integration?

In a focused attention paradigm, saccadic reaction time (SRT) to a visual target tends to be shorter when an auditory accessory stimulus is presented in close temporal and spatial proximity. Observed SRT reductions typically diminish as spatial disparity between the stimuli increases. Here a visual target LED (500 ms duration) was presented above or below the fixation point and a simultaneously presented auditory accessory (2 ms duration) could appear at the same or the opposite vertical position. SRT enhancement was about 35 ms in the coincident and 10 ms in the disparate condition. In order to further probe the audiovisual integration mechanism, in addition to the auditory non-target an auditory masker (200 ms duration) was presented before, simultaneous to, or after the accessory stimulus. In all interstimulus interval (ISI) conditions, SRT enhancement went down both in the coincident and disparate configuration, but this decrement was fairly stable across the ISI values. If multisensory integration solely relied on a feed-forward process, one would expect a monotonic decrease of the masker effect with increasing ISI in the backward masking condition. It is therefore conceivable that the relatively high-energetic masker causes a broad excitatory response of SC neurons. During this state, the spatial audio-visual information from multisensory association areas is fed back and merged with the spatially unspecific excitation pattern induced by the masker. Assuming that a certain threshold of activation has to be achieved in order to generate a saccade in the correct direction, the blurred joint output of noise and spatial audio-visual information needs more time to reach this threshold prolonging SRT to an audio-visual object.     

The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior

Physiological and behavioral studies in cat have shown that corticotectal influences play important roles in the information-processing capabilities of superior colliculus (SC) neurons. While corticotectal inputs from the anterior ectosylvian sulcus (AES) play a comparatively small role in the unimodal responses of SC neurons, they are particularly important in rendering these neurons capable of integrating information from different sensory modalities (e.g., visual and auditory). The present experiments examined the behavioral consequences of depriving SC neurons of AES inputs, and thereby compromising their ability to integrate visual and auditory information. Selective deactivation of a variety of other cortical areas (posterolateral lateral suprasylvian cortex, PLLS; primary auditory cortex, AI; or primary visual cortex, 17/18) served as controls. Cats were trained in a perimetry device to ignore a brief, low-intensity auditory stimulus but to orient toward and approach a nearthreshold visual stimulus (a light-emitting diode, LED) to obtain food. The LED was presented at different eccentricities either alone (unimodal) or combined with the auditory stimulus (multisensory). Subsequent deactivation of the AES, with focal injections of a local anesthetic, had no effect on responses to unimodal cues regardless of their location. However, it profoundly, though reversibly, altered orientation and approach to multisensory stimuli in contralateral space. The characteristic enhancement of these responses observed when an auditory cue was presented in spatial correspondence with the visual stimulus was significantly degraded. Similarly, the inhibitory effect of a spatially disparate auditory cue was significantly ameliorated. The observed effects were specific to AES deactivation, as similar effects were not obtained with deactivation of PLLS, AI or 17/18, or saline injections into the AES. These observations are consistent with postulates that specific cortical-midbrain interactions are essential for the synthesis of multisensory information in the SC, and for the orientation and localization behaviors that depend on this synthesis.     

Neuron-specific response characteristics predict the magnitude of multisensory integration

Multisensory neurons in the superior colliculus (SC) typically respond to combinations of stimuli from multiple modalities with enhancements and/or depressions in their activity. Although such changes in response have been shown to follow a predictive set of integrative principles, these principles fail to completely account for the full range of interactions seen throughout the SC population. In an effort to better define this variability, we sought to determine if there were additional features of the neuronal response profile that were predictive of the magnitude of the multisensory interaction. To do this, we recorded from 109 visual-auditory SC neurons while systematically manipulating stimulus intensity. Along with the previously described roles of space, time, and stimulus effectiveness, two features of a neuron's response profile were found to offer predictive value as to the magnitude of the multisensory interaction: spontaneous activity and the level of sensory responsiveness. Multisensory neurons with little or no spontaneous activity and weak sensory responses had the capacity to exhibit large response enhancements. Conversely, neurons with modest spontaneous activity and robust sensory responses exhibited relatively small response enhancements. Together, these results provide a better view into multisensory integration, and suggest substantial heterogeneity in the integrative characteristics of the multisensory SC population.     

The development of a dialogue between cortex and midbrain to integrate multisensory information

The anterior ectosylvian (AES) and rostral lateral suprasylvian (rLS) sulci send critical signals to multisensory superior colliculus (SC) neurons that enable them to integrate information from different senses. When either of these areas is temporarily deactivated in adult animals, the ability of SC neurons to integrate multisensory information and, thereby, enhance their responses to cross-modal stimuli is temporarily compromised. As a consequence, the ability to use cross-modal stimuli to enhance SC-mediated behavioral performance is also compromised. In contrast, removal of either one of these areas during early life has little effect on the development of multisensory processes in the SC or on SC-mediated multisensory behaviors and these animals seem very similar to normal controls. These observations suggest that there is considerable plasticity in these cortico-collicular systems during early life, with each area able to compensate for the early loss of the other. However, when both AES and rLS are removed early in life, there appears to be no compensation. The SC neurons now deal with sensory stimuli, even those embedded in multisensory complexes, as if they were there alone, precluding any SC-mediated behavioral benefit to cross-modal stimuli.     

Excitotoxic lesions of the superior colliculus preferentially impact multisensory neurons and multisensory integration

The superior colliculus (SC) plays an important role in integrating visual, auditory and somatosensory information, and in guiding the orientation of the eyes, ears and head. Previously we have shown that cats with unilateral SC lesions showed a preferential loss of multisensory orientation behaviors for stimuli contralateral to the lesion. Surprisingly, this behavioral loss was seen even under circumstances where the SC lesion was far from complete. To assess the physiological changes induced by these lesions, we employed single unit electrophysiological methods to record from individual neurons in both the intact and damaged SC following behavioral testing in two animals. In the damaged SC of these animals, multisensory neurons were preferentially reduced in incidence, comprising less than 25% of the sensory-responsive population (as compared with 49% on the control side). In those multisensory neurons that remained following the lesion, receptive fields were nearly twofold larger, and less than 25% showed normal patterns of multisensory integration, with those that did being found in areas outside of the lesion. These results strongly suggest that the multisensory behavioral deficits seen following SC lesions are the combined result of a loss of multisensory neurons and a loss of multisensory integration in those neurons that remain.     

A computational study of multisensory maturation in the superior colliculus (SC)

Multisensory neurons in cat SC exhibit significant postnatal maturation. The first multisensory neurons to appear have large receptive fields (RFs) and cannot integrate information across sensory modalities. During the first several months of postnatal life RFs contract, responses become more robust and neurons develop the capacity for multisensory integration. Recent data suggest that these changes depend on both sensory experience and active inputs from association cortex. Here, we extend a computational model we developed (Cuppini et al. in Front Integr Neurosci 22: 4-6, 2010) using a limited set of biologically realistic assumptions to describe how this maturational process might take place. The model assumes that during early life, cortical-SC synapses are present but not active and that responses are driven by non-cortical inputs with very large RFs. Sensory experience is modeled by a "training phase" in which the network is repeatedly exposed to modality-specific and cross-modal stimuli at different locations. Cortical-SC synaptic weights are modified during this period as a result of Hebbian rules of potentiation and depression. The result is that RFs are reduced in size and neurons become capable of responding in adult-like fashion to modality-specific and cross-modal stimuli.     

The co-occurrence of multisensory facilitation and cross-modal conflict in the human brain

Perceptual objects often comprise a visual and auditory signature that arrives simultaneously through distinct sensory channels, and cross-modal features are linked by virtue of being attributed to a specific object. Continued exposure to cross-modal events sets up expectations about what a given object most likely "sounds" like, and vice versa, thereby facilitating object detection and recognition. The binding of familiar auditory and visual signatures is referred to as semantic, multisensory integration. Whereas integration of semantically related cross-modal features is behaviorally advantageous, situations of sensory dominance of one modality at the expense of another impair performance. In the present study, magnetoencephalography recordings of semantically related cross-modal and unimodal stimuli captured the spatiotemporal patterns underlying multisensory processing at multiple stages. At early stages, 100 ms after stimulus onset, posterior parietal brain regions responded preferentially to cross-modal stimuli irrespective of task instructions or the degree of semantic relatedness between the auditory and visual components. As participants were required to classify cross-modal stimuli into semantic categories, activity in superior temporal and posterior cingulate cortices increased between 200 and 400 ms. As task instructions changed to incorporate cross-modal conflict, a process whereby auditory and visual components of cross-modal stimuli were compared to estimate their degree of congruence, multisensory processes were captured in parahippocampal, dorsomedial, and orbitofrontal cortices 100 and 400 ms after stimulus onset. Our results suggest that multisensory facilitation is associated with posterior parietal activity as early as 100 ms after stimulus onset. However, as participants are required to evaluate cross-modal stimuli based on their semantic category or their degree of congruence, multisensory processes extend in cingulate, temporal, and prefrontal cortices.     

Semantic congruence is a critical factor in multisensory behavioral performance

It has repeatedly been demonstrated that the presence of multiple cues in different sensory modalities can enhance behavioral performance by speeding responses, increasing accuracy, and/or improving stimulus detection. Despite an extensive knowledge base as to how the spatial, temporal, and physical (e.g., intensity) characteristics of multisensory stimuli influence such enhancements, little is known about the role of semantic or contextual congruence. Our hypothesis was that semantically congruent multisensory stimuli would result in enhanced behavioral performance, and that semantically incongruent multisensory stimuli would result in either no enhancement or a decrement in behavioral performance. The results from a redundant cue feature discrimination task clearly demonstrate that congruent cross-modal stimulation improves behavioral performance. This effect is specific to the multisensory stimuli, as no improvements are seen in the presence of redundant unimodal stimulus pairs. In contrast, incongruent stimulus pairs result in behavioral decrements for both multisensory and paired unimodal stimuli. These results highlight that in addition to such simple stimulus features as space, time and relative effectiveness, the semantic content of a multisensory stimulus plays a critical role in determining how it is processed by the nervous system.     

Disparity-selective neurons in area V4 of macaque monkeys

Area V4 is an intermediate stage of the ventral visual pathway providing major input to the final stages in the inferior temporal cortex (IT). This pathway is involved in the processing of shape, color, and texture. IT neurons are also sensitive to horizontal binocular disparity, suggesting that binocular disparity is processed along the ventral visual pathway. In the present study, we examined the processing of binocular disparity information by V4 neurons. We recorded responses of V4 neurons to binocularly disparate stimuli. A population of V4 neurons modified their responses according to changes of stimulus disparity; neither monocular responses nor eye movements could account for this modulation. Disparity-tuning curves were similar for different locations within a neuron's receptive field. Neighboring neurons recorded using a single electrode displayed similar disparity-tuning properties. These findings indicate that a population of V4 neurons is selective for binocular disparity, invariant for the position of the stimulus within the receptive field. The finding that V4 neurons with similar disparity selectivity are clustered suggests the existence of functional modules for disparity processing in V4.     

Neural mechanisms for synthesizing sensory information and producing adaptive behaviors

The ability to integrate information from different sensory systems is a fundamental characteristic of the brain. Because different bits of information are derived from different sensory channels, their synthesis markedly enhances the detection and identification of external stimuli. The neural substrate for such “multisensory” integration is provided by neurons that receive convergent input from two or more sensory modalities. Many such multisensory neurons are found in the superior colliculus (SC), a midbrain structure that plays a significant role in overt attentive and orientation behaviors. The various principles governing the integration of visual, auditory, and somatosensory inputs in SC neurons have been explored in several species. Thus far, the evidence suggests a remarkable conservation of integrative features during vertebrate evolution. One of the most robust of these principles is based on spatial relationships: a striking enhancement in activity is induced in a multisensory neuron when two different sensory stimuli (e.g., visual and auditory) are in spatial concordance, whereas a profound response depression can be induced when these cues are spatially discordant. The most extensive physiological observations have been made in cat, and in this species the same principles that have been shown to govern multisensory integration at the level of the individual SC neuron have also been shown to govern overt attentive and orientation responses to multisensory stimuli. Most surprising, however, is the critical role played by association (i.e. anterior ectosylvian) cortex in facilitating these midbrain processes. In the absence of the modulating corticotectal influences, multisensory SC neurons in cat are unable to integrate the different sensory cues converging upon them in an adult-like fashion, and are unable to mediate overt multisensory behaviors. This situation appears quite similar to that observed during early postnatal life. When multisensory SC neurons first appear, they are able to respond to multiple sensory inputs but are unable to synthesize these inputs to significantly enhance or degrade their responses. During ontogeny, individual multisensory neurons develop this capacity abruptly, but at very different ages, until the mature population condition is reached after several postnatal months. It appears likely that the abrupt onset of this capacity in any individual SC neuron reflects the maturation of inputs from anterior ectosylvian cortex. Presumably, the functional coupling of cortex with an individual SC neuron is essential to initiate and maintain that neuron’s capability for multisensory integration throughout its life.     

Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration

Convergence of inputs from different sensory modalities onto individual neurons is a phenomenon that occurs widely throughout the brain at many phyletic levels and appears to represent a basic neural mechanism by which an organism integrates complex environmental stimuli. In the present study, neurons in the superior colliculus (SC) were used as a model to examine how single neurons deal with simultaneous cues from different sensory modalities (e.g., visual, auditory, somatosensory). The functional result of multisensory convergence on an individual cell was determined by comparing the responses evoked from it by a combined-modality (multimodal) stimulus with those elicited by each (unimodal) component of that stimulus presented alone. Superior colliculus cells exhibited profound changes in their activity when individual sensory stimuli were combined. These "multisensory interactions" were found to be widespread among deep laminae cells and fell into one of two functional categories: response enhancement, characterized by a significant increase in the number of discharges evoked; and response depression, characterized by a significant decrease in the discharges elicited. Multisensory response interactions most often reflected a multiplicative, rather than summative, change in activity. Their absolute magnitude varied from cell to cell and, when stimulus conditions were altered, within the same cell. However, the percentage change of enhanced interactions was generally inversely related to the vigor of the responses that could be evoked by presenting each unimodal stimulus alone and suggest that the potential for response amplification was greatest when responses evoked by individual stimuli were weakest. The majority of cells exhibiting multi-sensory characteristics were demonstrated to have descending efferent projections and thus had access to premotor and motor areas of the brain stem and spinal cord involved in SC-mediated attentive and orientation behaviors. These data show that multisensory convergence provides the descending efferent cells of the SC with a dynamic response character. The responses of these cells and the SC-mediated behaviors that they underlie need not be immutably tied to the presence of any single stimulus, but can vary in response to the particular complex of stimuli present in the environment at any given moment.     

Receptive field dimensionality increases from the auditory midbrain to cortex

In the primary auditory cortex, spectrotemporal receptive fields (STRFs) are composed of multiple independent components that capture the processing of disparate stimulus aspects by any given neuron. The origin of these multidimensional stimulus filters in the central auditory system is unknown. To determine whether multicomponent STRFs emerge prior to the forebrain, we recorded from single neurons in the main obligatory station of the auditory midbrain, the inferior colliculus. By comparing results of different spike-triggered techniques, we found that the neural responses in the inferior colliculus can be accounted for by a single stimulus filter. This was observed for all temporal response patterns, from strongly phasic to tonic. Our results reveal that spectrotemporal stimulus encoding undergoes a fundamental transformation along the auditory neuraxis, with the emergence of multidimensional receptive fields beyond the auditory midbrain.     

Behavioral studies of auditory-visual spatial recognition and integration in rats

Rodents are useful animal models in the study of the molecular and cellular mechanisms underlying various neural functions. For studying behavioral properties associated with multisensory functions in rats, we measured the speed and accuracy of target detection by the reaction-time procedure. In the first experiment, we utilized simple two-alternative-choice tasks, in which spatial cues are visual or auditory modalities, and conducted a cross-modal transfer test in order to determine whether rats recognize amodal spatial information. Rats showed successful performance in the cross-modal transfer test and the speed to respond to sensory stimuli was constant under a rule-consistent condition despite the change in cue modality. In the second experiment, we developed audiovisual two-alternative-choice tasks, in which both auditory and visual stimuli were simultaneously presented but one of the two modalities was task-relevant, in order to determine whether the response to the sensory stimulation of one modality is enhanced by the stimulation of a different modality. If bimodal stimuli were spatially coincident, the speed for detecting the relevant stimulus was shortened and the extent of the effect was comparable to those in past studies of humans and other mammals. These results indicate the cross-modal spatial abilities of rats and our present paradigms may provide useful behavioral tasks for studying the neural bases of multisensory processing and integration in rats.     

Superior colliculus lesions preferentially disrupt multisensory orientation

The general involvement of the superior colliculus (SC) in orientation behavior and the striking parallels between the multisensory responses of SC neurons and overt orientation behaviors have led to assumptions that these neural and behavioral changes are directly linked. However, deactivation of two areas of cortex which also contain multisensory neurons, the anterior ectosylvian sulcus and rostral lateral suprasylvian sulcus have been shown to eliminate multisensory orientation behaviors, suggesting that this behavior may not involve the SC. To determine whether the SC contributes to this behavior, cats were tested in a multisensory (i.e. visual-auditory) orientation task before and after excitotoxic lesions of the SC. For unilateral SC lesions, modality-specific (i.e. visual or auditory) orientation behaviors had returned to pre-lesion levels after several weeks of recovery. In contrast, the enhancements and depressions in behavior normally seen with multisensory stimuli were severely compromised in the contralesional hemifield. No recovery of these behaviors was observed within the 6 month testing period. Immunohistochemical labeling of the SC revealed a preferential loss of parvalbumin-immunoreactive pyramidal neurons in the intermediate layers, a presumptive multisensory population that targets premotor areas of the brainstem and spinal cord. These results highlight the importance of the SC for multisensory behaviors, and suggest that the multisensory orientation deficits produced by cortical lesions are a result of the loss of cortical influences on multisensory SC neurons.