Low-level visual saliency does not predict change detection in natural scenes

Saliency models of eye guidance during scene perception suggest that attention is drawn to visually conspicuous areas having high visual salience. Despite such low-level visual processes controlling the allocation of attention, higher level information gained from scene knowledge may also control eye movements. This is supported by the findings of eye-tracking studies demonstrating that scene-inconsistent objects are often fixated earlier than their consistent counterparts. Using a change blindness paradigm, changes were made to objects that were either consistent or inconsistent with the scene and that had been measured as having high or low visual salience (according to objective measurements). Results showed that change detection speed and accuracy for objects with high visual salience did not differ from those having low visual salience. However, changes in scene-inconsistent objects were detected faster and with higher accuracy than those in scene-consistent objects for both high and low visually salient objects. We conclude that the scene-inconsistent change detection advantage is a true top-down effect and is not confounded by low-level visual factors and may indeed override such factors when viewing complex naturalistic scenes.     

Meaning and expected surfaces combine to guide attention during visual search in scenes

How do spatial constraints and meaningful scene regions interact to control overt attention during visual search for objects in real-world scenes? To answer this question, we combined novel surface maps of the likely locations of target objects with maps of the spatial distribution of scene semantic content. The surface maps captured likely target surfaces as continuous probabilities. Meaning was represented by meaning maps highlighting the distribution of semantic content in local scene regions. Attention was indexed by eye movements during the search for target objects that varied in the likelihood they would appear on specific surfaces. The interaction between surface maps and meaning maps was analyzed to test whether fixations were directed to meaningful scene regions on target-related surfaces. Overall, meaningful scene regions were more likely to be fixated if they appeared on target-related surfaces than if they appeared on target-unrelated surfaces. These findings suggest that the visual system prioritizes meaningful scene regions on target-related surfaces during visual search in scenes.     

Effects of scene inversion on change detection of targets matched for visual salience

This work examines how context may influence the detection of changes in flickering scenes. Each scene contained two changes that were matched for low-level visual salience. One of the changes was of high interest to the meaning of the scene, and the other was of lower interest. High-interest changes were more readily detected. To further examine the effects of contextual significance, we inverted the scene orientation to disrupt top-down effects of global context while controlling for contributions of visual salience. In other studies, inverting scene orientation has had inconsistent effects on detection of high-interest changes. However, this experiment demonstrated that inverting scene orientation significantly reduced the advantage for high-interest changes in comparison to lower-interest changes. Thus, scene context influences the deployment of attention and change-detection performance, and this top-down influence may be disrupted by scene inversion.     

An advantage for detecting dynamic targets in natural scenes

In the present study, we tested the extent to which observers use dynamic information to detect targets in natural scenes. For this purpose, we used composite stimuli in which target sequences were superimposed onto distractor sequences. We varied target visibility in the composite sequence, and the presence or absence of motion. Across four experiments, we found a dynamic advantage for target detection: Observers performed more accurately with dynamic than static target scenes. This advantage depended on the availability of target motion, irrespective of whether the target was upright or inverted in the image plane (Experiments 1-4). The magnitude of this advantage also depended on the availability of segmentation cues (Experiments 1 and 2) and on the distractors used (Experiments 2 and 4). Overall, the dynamic advantage reported extends previous work using isolated dynamic objects to more complex scenes.     

Shared cognitive mechanisms involved in the processing of scene texture and scene shape

Recent research has demonstrated that the parahippocampal place area represents both the shape and texture features of scenes, with the importance of each feature varying according to perceived scene category. Namely, shape features are predominately more diagnostic to the processing of artificial human–made scenes, while shape and texture are equally diagnostic in natural scene processing. However, to date little is known regarding the degree of interactivity or independence observed in the processing of these scene features. Furthermore, manipulating the scope of visual attention (i.e., globally vs. locally) when processing ensembles of multiple objects—stimuli that share a functional neuroanatomical link with scenes—has been shown to affect their cognitive visual representation. It remains unknown whether manipulating the scope of attention impacts scene processing in a similar manner. Using the well-established Garner speeded-classification behavioral paradigm, we investigated the influence of both feature diagnosticity and the scope of visual attention on potential interactivity or independence in the shape and texture processing of artificial human–made scenes. The results revealed asymmetric interference between scene shape and texture processing, with the more diagnostic feature (i.e., shape) interfering with the less diagnostic feature (i.e., texture), but not vice versa. Furthermore, this interference was attenuated and enhanced with more local and global visual processing strategies, respectively. These findings suggest that the scene shape and texture processing are mediated by shared cognitive mechanisms and that, although these representations are governed primarily via feature diagnosticity, they can nevertheless be influenced by the scope of visual attention.     

Semantic guidance of eye movements in real-world scenes

The perception of objects in our visual world is influenced by not only their low-level visual features such as shape and color, but also their high-level features such as meaning and semantic relations among them. While it has been shown that low-level features in real-world scenes guide eye movements during scene inspection and search, the influence of semantic similarity among scene objects on eye movements in such situations has not been investigated. Here we study guidance of eye movements by semantic similarity among objects during real-world scene inspection and search. By selecting scenes from the LabelMe object-annotated image database and applying latent semantic analysis (LSA) to the object labels, we generated semantic saliency maps of real-world scenes based on the semantic similarity of scene objects to the currently fixated object or the search target. An ROC analysis of these maps as predictors of subjects’ gaze transitions between objects during scene inspection revealed a preference for transitions to objects that were semantically similar to the currently inspected one. Furthermore, during the course of a scene search, subjects’ eye movements were progressively guided toward objects that were semantically similar to the search target. These findings demonstrate substantial semantic guidance of eye movements in real-world scenes and show its importance for understanding real-world attentional control.     

Equivalent noise characterization of human lightness constancy

A goal of visual perception is to provide stable representations of task-relevant scene properties (e.g. object reflectance) despite variation in task-irrelevant scene properties (e.g. illumination and reflectance of other nearby objects). To study such stability in the context of the perceptual representation of lightness, we introduce a threshold-based psychophysical paradigm. We measure how thresholds for discriminating the achromatic reflectance of a target object (task-relevant property) in rendered naturalistic scenes are impacted by variation in the reflectance functions of background objects (task-irrelevant property), using a two-alternative forced-choice paradigm in which the reflectance of the background objects is randomized across the two intervals of each trial. We control the amount of background reflectance variation by manipulating a statistical model of naturally occurring surface reflectances. For low background object reflectance variation, discrimination thresholds were nearly constant, indicating that observers’ internal noise determines threshold in this regime. As background object reflectance variation increases, its effects start to dominate performance. A model based on signal detection theory allows us to express the effects of task-irrelevant variation in terms of the equivalent noise, that is relative to the intrinsic precision of the task-relevant perceptual representation. The results indicate that although naturally occurring background object reflectance variation does intrude on the perceptual representation of target object lightness, the effect is modest – within a factor of two of the equivalent noise level set by internal noise.     

Competition and selection during visual processing of natural scenes and objects

When a visual scene, containing many discrete objects, is presented to our retinae, only a subset of these objects will be explicitly represented in visual awareness. The number of objects accessing short-term visual memory might be even smaller. Finally, it is not known to what extent "ignored" objects (those that do not enter visual awareness) will be processed--or recognized. By combining free recall, forced-choice recognition and visual priming paradigms for the same natural visual scenes and subjects, we were able to estimate these numbers, and provide insights as to the fate of objects that are not explicitly recognized in a single fixation. When presented for 250 ms with a scene containing 10 distinct objects, human observers can remember up to 4 objects with full confidence, and between 2 and 3 more when forced to guess. Importantly, the objects that the subjects consistently failed to report elicited a significant negative priming effect when presented in a subsequent task, suggesting that their identity was represented in high-level cortical areas of the visual system, before the corresponding neural activity was suppressed during attentional selection. These results shed light on neural mechanisms of attentional competition, and representational capacity at different levels of the human visual system.     

Impact of age‐related macular degeneration on object searches in realistic panoramic scenes

Background

This study investigated whether realistic immersive conditions with dynamic indoor scenes presented on a large, hemispheric panoramic screen covering 180° of the visual field improved the visual search abilities of participants with age‐related macular degeneration (AMD).

Method

Twenty‐one participants with AMD, 16 age‐matched controls and 16 young observers were included. Realistic indoor scenes were presented on a panoramic five metre diameter screen. Twelve different objects were used as targets. The participants were asked to search for a target object, shown on paper before each trial, within a room composed of various objects. A joystick was used for navigation within the scene views. A target object was present in 24 trials and absent in 24 trials. The percentage of correct detection of the target, the percentage of false alarms (that is, the detection of the target when it was absent), the number of scene views explored and the search time were measured.

Results

The search time was slower for participants with AMD than for the age‐matched controls, who in turn were slower than the young participants. The participants with AMD were able to accomplish the task with a performance of 75 per cent correct detections. This was slightly lower than older controls (79.2 per cent) while young controls were at ceiling (91.7 per cent). Errors were mainly due to false alarms resulting from confusion between the target object and another object present in the scene in the target‐absent trials.

Conclusion

The outcomes of the present study indicate that, under realistic conditions, although slower than age‐matched, normally sighted controls, participants with AMD were able to accomplish visual searches of objects with high accuracy.     

Change detection for objects on surfaces slanted in depth

Change detection for objects associated with a surface extended in depth might be more difficult than for a frontal surface if it is easier to shift attention within a frontal surface. On the other hand, previous research has shown that ground surfaces have a special role in organizing the 3-D layout of objects shown against scene backgrounds. In the current study, we examined whether a frontal background or a ground surface background would result in superior change detection performance using a change detection flicker paradigm. In the first experiment, we considered whether background slant affects change detection performance. In Experiment 2, we examined the effect of height in the image on change detection performance. In Experiment 3, we examined change detection performance on slanted ceiling surfaces. The results of these experiments indicate that change detection is more efficient on near-ground planes than on surfaces at intermediate slants or ceiling surfaces. This suggests that any superiority of frontal plane backgrounds in a change detection task may be equivalent to the superiority of a near-ground plane in organizing a scene, with the lowest level of performance occurring for surfaces that are not frontal but further from a ground surface orientation.     

A horizontal bias in human visual processing of orientation and its correspondence to the structural components of natural scenes

Many encoding mechanisms and processing strategies in the visual system appear to have evolved to better process the prevalent content in the visual world. Here we examine the relationship between the prevalence of natural scene content at different orientations and visual ability for detecting oriented natural scene content. Whereas testing with isolated gratings shows best performance at horizontal and vertical (the oblique effect), we report that when tested with natural scene content, performance is best at obliques and worst at horizontal (the horizontal effect). The present analysis of typical natural scenes shows that the prevalence of natural scene content matches the inverse of this horizontal effect pattern with most scene content at horizontal, next most at vertical, and least at obliques. We suggest that encoding of orientation may have evolved to accommodate the anisotropy in natural scene content by perceptually discounting the most prevalent oriented content in a scene, thereby increasing the relative salience of objects and other content in a scene when viewed against a typical natural background.     

How direction of illumination affects visually perceived surface roughness

We examined visual estimation of surface roughness using random, computer-generated, three-dimensional (3D) surfaces rendered under a mixture of diffuse lighting and a punctate source. The angle between the tangent to the plane containing the surface texture and the direction to the punctate source was varied from 50 to 70 deg across lighting conditions. Observers were presented with pairs of surfaces under different lighting conditions and indicated which 3D surface appeared rougher. Surfaces were viewed either in isolation or in scenes with added objects whose shading, cast shadows, and specular highlights provided information about the spatial distribution of illumination. All observers perceived surfaces to be markedly rougher with decreasing illuminant angle. Performance in scenes with added objects was no closer to constant than that in scenes without added objects. We identified four novel cues that are valid cues to roughness under any single lighting condition but that are not invariant under changes in lighting condition. We modeled observers' deviations from roughness constancy as a weighted linear combination of these "pseudocues" and found that they account for a substantial amount of observers' systematic deviations from roughness constancy with changes in lighting condition.     

Visual scene memory and the guidance of saccadic eye movements.

An unresolved question is how much information can be remembered from visual scenes when they are inspected by saccadic eye movements. Subjects used saccadic eye movements to scan a computer-generated scene, and afterwards, recalled as many objects as they could. Scene memory was quite good: it improved with display duration, it persisted over time long after the display was removed, and it continued to accumulate with additional viewings of the same display (Melcher, D. (2001) The persistance of memory for scenes. Nature 412, 401). The occurrence of saccadic eye movements was important to ensure good recall performance, even though subjects often recalled non-fixated objects. Inter-saccadic intervals increased with display duration, showing an influence of duration on global scanning strategy. The choice of saccadic target was predicted by a Random Selection with Distance Weighting (RSDW) model, in which the target for each saccade is selected at random from all available objects, weighted according to distance from fixation, regardless of which objects had previously been fixated. The results show that the visual memory that was reflected in the recall reports was not utilized for the immediate decision about where to look in the scene. Visual memory can be excellent, but it is not always reflected in oculomotor measures, perhaps because the cost of rapid on-line memory retrieval is too great.     

Estimating changes in lighting direction in binocularly viewed three-dimensional scenes

We examine human ability to detect changes in scene lighting. Thirteen observers viewed three-dimensional rendered scenes stereoscopically. Each scene consisted of a randomly generated three-dimensional "Gaussian bump" surface rendered under a combination of collimated and diffuse light sources. During each trial, the collimated source underwent a small, quick change of position in one of four directions. The observer's task was to classify the direction of the lighting change. All observers were above chance in performing the task. We developed a model that combined two sources of information, a shape map and a shading map, to predict lighting change direction. We used this model to predict patterns of errors both across observers and across scenes differing in shape. We found that errors in estimating lighting direction were primarily the result of errors in representing surface shape. We characterized the surface features that affected performance in the classification task.     

Object co-occurrence serves as a contextual cue to guide and facilitate visual search in a natural viewing environment

There is accumulating evidence that scene context can guide and facilitate visual search (e.g., A. Torralba, A. Oliva, M. S. Castelhano, & J. M. Henderson, 2006). Previous studies utilized stimuli of restricted size, a fixed head position, and context defined by the global spatial configuration of the scene. Thus, it is unknown whether similar effects generalize to natural viewing environments and to context defined by local object co-occurrence. Here, with a mobile eye tracker, we investigated the effects of object co-occurrence on search performance under naturalistic conditions. Observers searched for low-visibility target objects on tables cluttered with everyday objects. Targets were either located adjacent to larger, more visible "cue" objects that they regularly co-occurred in natural scenes (expected condition) or elsewhere in the display, surrounded by unrelated objects (unexpected condition). Mean search times were shorter for targets at expected locations as compared to unexpected locations. Additionally, context guided eye movements, as more fixations were directed toward cue objects than other non-target objects, particularly when the cue was contextually relevant to the current search target. These results could not be accounted for by image saliency models. Thus, we conclude that object co-occurrence can serve as a contextual cue to facilitate search and guide eye movements in natural environments.     

Spatiotopic and retinotopic memory in the context of natural images

Neural responses throughout the visual cortex encode stimulus location in a retinotopic (i.e., eye-centered) reference frame, and memory for stimulus position is most precise in retinal coordinates. Yet visual perception is spatiotopic: objects are perceived as stationary, even though eye movements cause frequent displacement of their location on the retina. Previous studies found that, after a single saccade, memory of retinotopic locations is more accurate than memory of spatiotopic locations. However, it is not known whether various aspects of natural viewing affect the retinotopic reference frame advantage. We found that the retinotopic advantage may in part depend on a retinal afterimage, which can be effectively nullified through backwards masking. Moreover, in the presence of natural scenes, spatiotopic memory is more accurate than retinotopic memory, but only when subjects are provided sufficient time to process the scene before the eye movement. Our results demonstrate that retinotopic memory is not always more accurate than spatiotopic memory and that the fidelity of memory traces in both reference frames are sensitive to the presence of contextual cues.     

Does disruption of a scene impair change detection?

When we view a scene, we generally feel that we have a rich representation of that scene. Recent research has shown, however, that we are unable to detect relatively large changes in scenes, which suggests an inability to retain the visual details from one scene view to the next. In the present study, we investigated whether we can retain and make use of global and semantic information from a scene in order to efficiently detect changes from one scene to the next. Results indicated that change detection was practically independent of scene disruption with one exception. Better performance in the meaningful scenes was observed only in the whole-scene presentation condition where the participants knew that the stimulus was extracted from the meaningful scene.     

Learning where to direct gaze during change detection

Where do observers direct their attention in complex scenes? Previous work on the cognitive control of fixation patterns in natural environments suggests that subjects must learn where to direct attention and gaze. We examined this question in the context of a change blindness paradigm, where some objects were more likely to undergo a change in orientation than others. The experiments revealed that observers are capable of learning the frequency with which objects undergo a change, and that this learning is manifested in the distribution of gaze among objects in the scene, as well as in the reaction time for detecting visual changes, and the frequency of localizing changing objects. However, observers were much less sensitive to the conditional probability of a second feature, border color, predicting a change in orientation. We conclude that striking demonstrations of change blindness may reflect not only the constraints of attention and working memory, but also what observers have learnt about what information to attend and select for storage during the task of change detection. Such exploitation of the frequency of change suggests that gaze allocation is sensitive to the probabilistic structure of the environment.     

The importance of peripheral vision when searching 3D real-world scenes: A gaze-contingent study in virtual reality

Visual search in natural scenes is a complex task relying on peripheral vision to detect potential targets and central vision to verify them. The segregation of the visual fields has been particularly established by on-screen experiments. We conducted a gaze-contingent experiment in virtual reality in order to test how the perceived roles of central and peripheral visions translated to more natural settings. The use of everyday scenes in virtual reality allowed us to study visual attention by implementing a fairly ecological protocol that cannot be implemented in the real world. Central or peripheral vision was masked during visual search, with target objects selected according to scene semantic rules. Analyzing the resulting search behavior, we found that target objects that were not spatially constrained to a probable location within the scene impacted search measures negatively. Our results diverge from on-screen studies in that search performances were only slightly affected by central vision loss. In particular, a central mask did not impact verification times when the target was grammatically constrained to an anchor object. Our findings demonstrates that the role of central vision (up to 6 degrees of eccentricities) in identifying objects in natural scenes seems to be minor, while the role of peripheral preprocessing of targets in immersive real-world searches may have been underestimated by on-screen experiments.     

Surface color perception in three-dimensional scenes

Researchers studying surface color perception have typically used stimuli that consist of a small number of matte patches (real or simulated) embedded in a plane perpendicular to the line of sight (a "Mondrian," Land & McCann, 1971). Reliable estimation of the color of a matte surface is a difficult if not impossible computational problem in such limited scenes (Maloney, 1999). In more realistic, three-dimensional scenes the difficulty of the problem increases, in part, because the effective illumination incident on the surface (the light field) now depends on surface orientation and location. We review recent work in multiple laboratories that examines (1) the degree to which the human visual system discounts the light field in judging matte surface lightness and color and (2) what illuminant cues the visual system uses in estimating the flow of light in a scene.