Interaction of visual prior constraints

The visual system relies on two types of information to interpret a visual scene: the cues that can be extracted from the retinal images and prior constraints that are used to disambiguate the scene. Many studies have looked at how multiple visual cues are combined. We examined the interaction of multiple prior constraints. The particular constraints studied here are assumptions the observer makes concerning the location of the light source (for the shading cue to depth) and the orientation of a surface (for depth based on image contours). The reliability of each of the two cues was manipulated by changing the contrast of different parts of the stimuli. We developed a model based on elements of Bayesian decision theory that permitted us to track the weights applied to each of the prior constraints as a function of the cue reliabilities. The results provided evidence that prior constraints behave just like visual cues to depth: cues with more reliable information have higher weight attributed to their corresponding prior constraint.     

Number and density discrimination rely on a common metric: Similar psychophysical effects of size, contrast, and divided attention

While observers are adept at judging the density of elements (e.g., in a random-dot image), it has recently been proposed that they also have an independent visual sense of number. To test the independence of number and density discrimination, we examined the effects of manipulating stimulus structure (patch size, element size, contrast, and contrast-polarity) and available attentional resources on both judgments. Five observers made a series of two-alternative, forced-choice discriminations based on the relative numerosity/density of two simultaneously presented patches containing 16-1,024 Gaussian blobs. Mismatches of patch size and element size (across reference and test) led to bias and reduced sensitivity in both tasks, whereas manipulations of contrast and contrast-polarity had varied effects on observers, implying differing strategies. Nonetheless, the effects reported were consistent across density and number judgments, the only exception being when luminance cues were made available. Finally, density and number judgment were similarly impaired by attentional load in a dual-task experiment. These results are consistent with a common underlying metric to density and number judgments, with the caveat that additional cues may be exploited when they are available.     

Reaching for visual cues to depth: the brain combines depth cues differently for motor control and perception

Vision provides a number of cues about the three-dimensional (3D) layout of objects in a scene that could be used for planning and controlling goal-directed behaviors such as pointing, grasping, and placing objects. An emerging consensus from the perceptual work is that the visual brain is a near-optimal Bayesian estimator of object properties, for example, by integrating cues in a way that accounts for differences in their reliability. We measured how the visuomotor system integrates binocular and monocular cues to 3D surface orientation to guide the placement of objects on a slanted surface. Subjects showed qualitatively similar results to those found in perceptual studies--they gave more weight to binocular cues at low slants and more weight to monocular cues like texture at high slants. We compared subjects' performance in the visuomotor task with their performance on matched perceptual tasks that required an observer to estimate the same 3D surface properties needed to control the motor behavior. The relative influence of binocular and monocular cues changed in qualitatively the same way across stimulus conditions in the two types of task; however, subjects gave significantly more weight to binocular cues for controlling hand movements than for making explicit perceptual judgments in these tasks. Thus, the brain changes how it integrates visual cues based not only on the information content of stimuli, but also on the task for which the information is used.     

Ideal observer perturbation analysis reveals human strategies for inferring surface orientation from texture

Optical texture patterns contain three quasi-independent cues to planar surface orientation: perspective scaling, projective foreshortening and density. The purpose of this work was to estimate the perceptual weights assigned to these texture cues for discriminating surface orientation and to measure the visual system's reliance on an isotropy assumption in interpreting foreshortening information. A novel analytical technique is introduced which takes advantage of the natural cue perturbations inherent in stochastic texture stimuli to estimate cue weights and measure the influence of an isotropy assumption. Ideal observers were derived which compute the exact information content of the different texture cues in the stimuli used in the experiments and which either did or did not rely on an assumption of surface texture isotropy. Simulations of the ideal observers using the same stimuli shown to subjects in a slant discrimination task provided trial-by-trial estimates of the natural cue perturbations which were inherent in the stimuli. By back-correlating subjects' judgments with the different ideal observer estimates, we were able to estimate both the weights given to each cue by subjects and the strength of subjects' prior assumptions of isotropy. In all of the conditions tested, we found that subjects relied primarily on the foreshortening cue. A small, but significant weight was given to scaling information and no significant weight was given to density information. In conditions in which the surface textures deviated from isotropy by random amounts from stimulus to stimulus, subject judgements correlated well with the estimates of an ideal observer which incorrectly assumed surface texture isotropy. This correlation was not complete, however, suggesting that a soft form of the isotropy constraint was used. Moreover, the correlation was significantly lower for textures containing higher-order information about surface orientation (skew of rectangular texture elements). The results of the analysis clearly implicate texture foreshortening as a primary cue for perceiving surface slant from texture and suggest that the visual system incorporates a strong, though not complete, bias to interpret surface textures as isotropic in its inference of surface slant from texture. They further suggest that local texture skew, when available in an image, contributes significantly to perceptual estimates of surface orientation.     

Optimal integration of texture and motion cues to depth

We report the results of a depth-matching experiment in which subjects were asked to adjust the height of an ellipse until it matched the depth of a simulated cylinder defined by texture and motion cues. In one-third of the trials the shape of the cylinder was primarily given by motion information, in another one-third of the trials it was given by texture information, and on the remaining trials it was given by both sources of information. Two optimal cue combination models are described where optimality is defined in terms of Bayesian statistics. The parameter values of the models are set based on subjects' responses on trials when either the motion cue or the texture cue was informative. These models provide predictions of subjects' responses on trials when both cues were informative. The results indicate that one of the optimal models provides a good fit to the subjects' data, and the second model provides an exceptional fit. Because the predictions of the optimal models closely match the experimental data, we conclude that observers' cue-combination strategies are indeed optimal, at least under the conditions studied here.     

Necessary but not sufficient: motion perception is required for perceiving biological motion

Researchers have argued that biological motion perception from point-light animations is resolved from stationary form information. To determine whether motion is required for biological motion perception, we measured discrimination thresholds at isoluminance. Whereas simple direction discriminations falter at isoluminance, biological motion perception fails entirely. However, when performance is measured as a function of contrast, it is apparent that biological motion is contrast-dependent, while direction discriminations are contrast invariant. Our results are evidence that biological motion perception requires intact motion perception, but is also mediated by a secondary mechanism that may be the integration of form and motion, or the computation of higher-order motion cues.     

A reduction in stimulus duration can improve wavelength discriminations mediated by short-wave cones.

Virtually all visual discriminations become less accurate when either the luminance or the duration of the stimulus is reduced. An exception is found for wavelength discriminations near 460 nm, where an increase in either luminance or duration can cause the threshold to rise. For flashes of 100 msec or less, the critical variable is the total energy of the flash (i.e. the product of retinal illuminance and flash duration), and wavelength discrimination is optimal at an intermediate value; higher stimulus energy causes discrimination to deteriorate. To explain these findings we suppose that discrimination in this region of the spectrum is mediated by a channel that draws opposed signals from the short-wavelength cones and from some combination of the middle- and long-wavelength cones, and that high stimulus energies cause saturation of this channel.     

Top-down influences on visual attention during listening are modulated by observer sex

In conversation, women have a small advantage in decoding non-verbal communication compared to men. In light of these findings, we sought to determine whether sex differences also existed in visual attention during a related listening task, and if so, if the differences existed among attention to high-level aspects of the scene or to conspicuous visual features. Using eye-tracking and computational techniques, we present direct evidence that men and women orient attention differently during conversational listening. We tracked the eyes of 15 men and 19 women who watched and listened to 84 clips featuring 12 different speakers in various outdoor settings. At the fixation following each saccadic eye movement, we analyzed the type of object that was fixated. Men gazed more often at the mouth and women at the eyes of the speaker. Women more often exhibited “distracted” saccades directed away from the speaker and towards a background scene element. Examining the multi-scale center-surround variation in low-level visual features (static: color, intensity, orientation, and dynamic: motion energy), we found that men consistently selected regions which expressed more variation in dynamic features, which can be attributed to a male preference for motion and a female preference for areas that may contain nonverbal information about the speaker. In sum, significant differences were observed, which we speculate arise from different integration strategies of visual cues in selecting the final target of attention. Our findings have implications for studies of sex in nonverbal communication, as well as for more predictive models of visual attention.     

Asymmetries in the direction of saccades during perception of scenes and fractals: Effects of image type and image features

The direction in which people tend to move their eyes when inspecting images can reveal the different influences on eye guidance in scene perception, and their time course. We investigated biases in saccade direction during a memory-encoding task with natural scenes and computer-generated fractals. Images were rotated to disentangle egocentric and image-based guidance. Saccades in fractals were more likely to be horizontal, regardless of orientation. In scenes, the first saccade often moved down and subsequent eye movements were predominantly vertical, relative to the scene. These biases were modulated by the distribution of visual features (saliency and clutter) in the scene. The results suggest that image orientation, visual features and the scene frame-of-reference have a rapid effect on eye guidance.     

What’s “up”? Working memory contents can bias orientation processing

We explored the interaction between the processing of a low-level visual feature such as orientation and the contents of working memory (WM). In a first experiment, participants memorized the orientation of a Gabor patch and performed two subsequent orientation discriminations during the retention interval. The WM stimulus exerted a consistent repulsive effect on the discrimination judgments: participants were more likely to report that the discrimination stimulus was rotated clockwise compared to the oblique after being presented with a stimulus that was tilted anti-clockwise from the oblique. A control condition where participants attended to the Gabor patch but did not memorize it, showed a much reduced effect. The repulsive effect was stable across the two discriminations in the memory condition, but not in the control condition, where it decayed at the second discrimination. In a second experiment, we showed that the greater interference observed in the WM condition cannot be explained by a difference in cognitive demands between the WM and the control condition. We conclude that WM contents can bias perception: the effect of WM interference is of a visual nature, can last over delays of several seconds and is not disrupted by the processing of intervening visual stimuli during the retention period.     

Experience-dependent visual cue integration based on consistencies between visual and haptic percepts

We study the hypothesis that observers can use haptic percepts as a standard against which the relative reliabilities of visual cues can be judged, and that these reliabilities determine how observers combine depth information provided by these cues. Using a novel visuo-haptic virtual reality environment, subjects viewed and grasped virtual objects. In Experiment 1, subjects were trained under motion relevant conditions, during which haptic and visual motion cues were consistent whereas haptic and visual texture cues were uncorrelated, and texture relevant conditions, during which haptic and texture cues were consistent whereas haptic and motion cues were uncorrelated. Subjects relied more on the motion cue after motion relevant training than after texture relevant training, and more on the texture cue after texture relevant training than after motion relevant training. Experiment 2 studied whether or not subjects could adapt their visual cue combination strategies in a context-dependent manner based on context-dependent consistencies between haptic and visual cues. Subjects successfully learned two cue combination strategies in parallel, and correctly applied each strategy in its appropriate context. Experiment 3, which was similar to Experiment 1 except that it used a more naturalistic experimental task, yielded the same pattern of results as Experiment 1 indicating that the findings do not depend on the precise nature of the experimental task. Overall, the results suggest that observers can involuntarily compare visual and haptic percepts in order to evaluate the relative reliabilities of visual cues, and that these reliabilities determine how cues are combined during three-dimensional visual perception.     

Integration of sensory evidence in motion discrimination

To make perceptual judgments, the brain must decode the responses of sensory cortical neurons. The direction of visual motion is represented by the activity of direction-selective neurons. Because these neurons are often broadly tuned and their responses are inherently variable, the brain must appropriately integrate their responses to infer the direction of motion reliably. The optimal integration strategy is task dependent. For coarse direction discriminations, neurons tuned to the directions of interest provide the most reliable information, but for fine discriminations, neurons with preferred directions displaced away from the target directions are more informative. We measured coarse and fine direction discriminations with random-dot stimuli. Unknown to the observers, we added subthreshold motion signals of different directions to perturb the responses of different groups of direction-selective neurons. The pattern of biases induced by subthreshold signals of different directions indicates that subjects' choice behavior relied on the activity of neurons with a wide range of preferred directions. For coarse discriminations, observers' judgments were most strongly determined by neurons tuned to the target directions, but for fine discriminations, neurons with displaced preferred directions had the largest influence. We conclude that perceptual decisions rely on a population decoding strategy that takes the statistical reliability of sensory responses into account.     

How long do intrinsic and extrinsic visual cues take to exert their effect on the perceptual upright?

We determined the amount of time it took for intrinsic and extrinsic visual cues to determine the perceptual upright. The perceptual upright was measured using a probe, the identity of which depended on its perceived orientation (the Oriented Character Recognition Test). A visual background that filled the field of view and contained both intrinsic and extrinsic cues was presented in different orientations and for presentation times of between 50 and 500 ms followed by a mask. The contribution of each class of cue was identified by exploiting their different degrees of ambiguity. Intrinsic cues include scene structure (e.g., walls, floor and ceiling of an indoor scene) which indicates four potential up directions, and the horizon which indicates two possibilities. Extrinsic cues, which rely on information not in the image such as a surface acting as a support structure for an object, signal the direction of up unambiguously. The contribution of each class of visual cue could thus be identified from the number of cycles its effect showed as the background was presented in all orientations round the clock. While the more high-level extrinsic cues to up exerted a larger influence on the perceptual upright than the intrinsic cues, the magnitude of each cue’s effect increased with presentation time at approximately the same rate with a time constant of about 60 ms. This finding poses a challenge for bottom-up theories of scene perception and suggests that low-level and high-level information are processed in parallel at least insofar as they indicate orientation.     

Combining cues to judge distance and direction in an immersive virtual reality environment

When we move, the visual direction of objects in the environment can change substantially. Compared with our understanding of depth perception, the problem the visual system faces in computing this change is relatively poorly understood. Here, we tested the extent to which participants’ judgments of visual direction could be predicted by standard cue combination rules. Participants were tested in virtual reality using a head-mounted display. In a simulated room, they judged the position of an object at one location, before walking to another location in the room and judging, in a second interval, whether an object was at the expected visual direction of the first. By manipulating the scale of the room across intervals, which was subjectively invisible to observers, we put two classes of cue into conflict, one that depends only on visual information and one that uses proprioceptive information to scale any reconstruction of the scene. We find that the sensitivity to changes in one class of cue while keeping the other constant provides a good prediction of performance when both cues vary, consistent with the standard cue combination framework. Nevertheless, by comparing judgments of visual direction with those of distance, we show that judgments of visual direction and distance are mutually inconsistent. We discuss why there is no need for any contradiction between these two conclusions.     

Do humans optimally integrate stereo and texture information for judgments of surface slant?

An optimal linear system for integrating visual cues to 3D surface geometry weights cues in inverse proportion to their uncertainty. The problem of integrating texture and stereo information for judgments of planar surface slant provides a strong test of optimality in human perception. Since the accuracy of slant from texture judgments changes by an order of magnitude from low to high slants, optimality predicts corresponding changes in cue weights as a function of surface slant. Furthermore, since humans show significant individual differences in their abilities to use both texture and stereo information for judgments of 3D surface geometry, the problem admits the stronger test that individual differences in subjects' thresholds for discriminating slant from the individual cues should predict individual differences in cue weights. We tested both predictions by measuring slant discrimination thresholds and stereo/texture cue weights as a function of surface slant for multiple subjects. The results bear out both predictions of optimality, with the exception of an apparent slight under-weighting of texture information. This may be accounted for by factors specific to the stimuli used to isolate stereo information in the experiments. Taken together, the results are consistent with the hypothesis that humans optimally combine the two cues to surface slant, with cue weights proportional to the subjective reliability of the cues.     

Slant from texture and disparity cues: optimal cue combination

How does the visual system combine information from different depth cues to estimate three-dimensional scene parameters? We tested a maximum-likelihood estimation (MLE) model of cue combination for perspective (texture) and binocular disparity cues to surface slant. By factoring the reliability of each cue into the combination process, MLE provides more reliable estimates of slant than would be available from either cue alone. We measured the reliability of each cue in isolation across a range of slants and distances using a slant-discrimination task. The reliability of the texture cue increases as |slant| increases and does not change with distance. The reliability of the disparity cue decreases as distance increases and varies with slant in a way that also depends on viewing distance. The trends in the single-cue data can be understood in terms of the information available in the retinal images and issues related to solving the binocular correspondence problem. To test the MLE model, we measured perceived slant of two-cue stimuli when disparity and texture were in conflict and the reliability of slant estimation when both cues were available. Results from the two-cue study indicate, consistent with the MLE model, that observers weight each cue according to its relative reliability: Disparity weight decreased as distance and |slant| increased. We also observed the expected improvement in slant estimation when both cues were available. With few discrepancies, our data indicate that observers combine cues in a statistically optimal fashion and thereby reduce the variance of slant estimates below that which could be achieved from either cue alone. These results are consistent with other studies that quantitatively examined the MLE model of cue combination. Thus, there is a growing empirical consensus that MLE provides a good quantitative account of cue combination and that sensory information is used in a manner that maximizes the precision of perceptual estimates.     

Face-gender discrimination is possible in the near-absence of attention

The attentional cost associated with the visual discrimination of the gender of a face was investigated. Participants performed a face-gender discrimination task either alone (single-task) or concurrently (dual-task) with a known attentional demanding task (5-letter T/L discrimination). Overall performance on face-gender discrimination suffered remarkably little under the dual-task condition compared to the single-task condition. Similar results were obtained in experiments that controlled for potential training effects or the use of low-level cues in this discrimination task. Our results provide further evidence against the notion that only low-level representations can be accessed outside the focus of attention.     

Stimulus uncertainty and perceptual learning: Similar principles govern auditory and visual learning

We examined the impact of variability in speech stimuli on improvement of general performance and on accessibility to low-level information as a function of practice. Listeners had to discriminate between two similar words in noise in two configurations that differed only in their low-level binaural information, which was either null or maximal. The difference in performance quantifies the use of binaural low-level information. These configurations were presented in three training protocols: in separate blocks; in a consistently interleaved manner; and in a randomly mixed manner. The first protocol enabled optimal use of the low-level binaural cues already at the first training session. The second, consistently interleaved protocol required more than one training session to reach the same performance. The final, mixed protocol did not enable optimal use of the low-level cues even after multi-session training. Interestingly, training with the first two protocols transferred to the mixed one. These results are in line with recent findings in the visual modality. In both modalities, the effects of variability on learning can be explained by the introduction of obstructions to a search mechanism going down along the sensory processing hierarchy, as suggested by the Reverse Hierarchy Theory.     

Experience-dependent visual cue recalibration based on discrepancies between visual and haptic percepts

We studied the hypothesis that observers can recalibrate their visual percepts when visual and haptic (touch) cues are discordant and the haptic information is judged to be reliable. Using a novel visuo-haptic virtual reality environment, we conducted a set of experiments in which subjects interacted with scenes consisting of two fronto-parallel surfaces. Subjects judged the distance between the two surfaces based on two perceptual cues: a visual stereo cue obtained when viewing the scene binocularly and a haptic cue obtained when subjects grasped the two surfaces between their thumb and index fingers. Visual and haptic cues regarding the scene were manipulated independently so that they could either be consistent or inconsistent. Experiment 1 explored the effect of visuo-haptic inconsistencies on depth-from-stereo estimates. Our findings suggest that when stereo and haptic cues are inconsistent, subjects recalibrate their interpretations of the visual stereo cue so that depth-from-stereo percepts are in greater agreement with depth-from-haptic percepts. In Experiment 2 the visuo-haptic discrepancy took a different form when the two surfaces were near the subject than when they were far from the subject. The results indicate that subjects recalibrated their interpretations of the stereo cue in a context-sensitive manner that depended on viewing distance, thereby making them more consistent with depth-from-haptic estimates at all viewing distances. Together these findings suggest that observers' visual and haptic percepts are tightly coupled in the sense that haptic percepts provide a standard to which visual percepts can be recalibrated when the visual percepts are deemed to be erroneous.     

Cortical area V4 is critical for certain texture discriminations, but this effect is not dependent on attention

This study examined the question of which features of a complex grouping discrimination make it vulnerable to permanent elimination by V4 lesions. We first verified that the line element grouping discrimination, which we previously reported to be devastated by V4 lesions, was similarly affected in the monkeys of this study. The permanence of the deficit was established by mapping its visual field distribution and then testing this discrimination for an extended period at a locus on the border of the deficit. Also, a staircase procedure was used to provide the monkey with within session instruction in the grouping discrimination, but this did not improve V4 lesion performance. Grouping was then compared with several discriminations that shared some features with it, but which were found not to be permanently eliminated by V4 lesions. This comparison suggested that grouping (rather than segmentation or response to a single element) was one feature that made the discrimination vulnerable, a second was the similarity in shape of the texture elements to be grouped. Finally, we tested visual crowding, a property of peripheral vision that is thought to reflect neuronal interactions early in visual cortex, possibly in area V1, and found no effect of V4 lesions. A control experiment with human observers tested whether the elimination of grouping by V4 lesions might be due to an alteration of attention, but found no evidence to support this hypothesis. These results show that severe disruption of texture discriminations by V4 lesions depends on both the nature of the discrimination and the type of texture elements involved, but does not necessarily involve the disruption of attention.