Semantic guidance of eye movements in real-world scenes

The perception of objects in our visual world is influenced by not only their low-level visual features such as shape and color, but also their high-level features such as meaning and semantic relations among them. While it has been shown that low-level features in real-world scenes guide eye movements during scene inspection and search, the influence of semantic similarity among scene objects on eye movements in such situations has not been investigated. Here we study guidance of eye movements by semantic similarity among objects during real-world scene inspection and search. By selecting scenes from the LabelMe object-annotated image database and applying latent semantic analysis (LSA) to the object labels, we generated semantic saliency maps of real-world scenes based on the semantic similarity of scene objects to the currently fixated object or the search target. An ROC analysis of these maps as predictors of subjects’ gaze transitions between objects during scene inspection revealed a preference for transitions to objects that were semantically similar to the currently inspected one. Furthermore, during the course of a scene search, subjects’ eye movements were progressively guided toward objects that were semantically similar to the search target. These findings demonstrate substantial semantic guidance of eye movements in real-world scenes and show its importance for understanding real-world attentional control.     

Idiosyncratic characteristics of saccadic eye movements when viewing different visual environments.

Eye position was recorded in different viewing conditions to assess whether the temporal and spatial characteristics of saccadic eye movements in different individuals are idiosyncratic. Our aim was to determine the degree to which oculomotor control is based on endogenous factors. A total of 15 naive subjects viewed five visual environments: (1) The absence of visual stimulation (i.e. a dark room); (2) a repetitive visual environment (i.e. simple textured patterns); (3) a complex natural scene; (4) a visual search task; and (5) reading text. Although differences in visual environment had significant effects on eye movements, idiosyncrasies were also apparent. For example, the mean fixation duration and size of an individual's saccadic eye movements when passively viewing a complex natural scene covaried significantly with those same parameters in the absence of visual stimulation and in a repetitive visual environment. In contrast, an individual's spatio-temporal characteristics of eye movements during active tasks such as reading text or visual search covaried together, but did not correlate with the pattern of eye movements detected when viewing a natural scene, simple patterns or in the dark. These idiosyncratic patterns of eye movements in normal viewing reveal an endogenous influence on oculomotor control. The independent covariance of eye movements during different visual tasks shows that saccadic eye movements during active tasks like reading or visual search differ from those engaged during the passive inspection of visual scenes.     

Meaning and expected surfaces combine to guide attention during visual search in scenes

How do spatial constraints and meaningful scene regions interact to control overt attention during visual search for objects in real-world scenes? To answer this question, we combined novel surface maps of the likely locations of target objects with maps of the spatial distribution of scene semantic content. The surface maps captured likely target surfaces as continuous probabilities. Meaning was represented by meaning maps highlighting the distribution of semantic content in local scene regions. Attention was indexed by eye movements during the search for target objects that varied in the likelihood they would appear on specific surfaces. The interaction between surface maps and meaning maps was analyzed to test whether fixations were directed to meaningful scene regions on target-related surfaces. Overall, meaningful scene regions were more likely to be fixated if they appeared on target-related surfaces than if they appeared on target-unrelated surfaces. These findings suggest that the visual system prioritizes meaningful scene regions on target-related surfaces during visual search in scenes.     

Object co-occurrence serves as a contextual cue to guide and facilitate visual search in a natural viewing environment

There is accumulating evidence that scene context can guide and facilitate visual search (e.g., A. Torralba, A. Oliva, M. S. Castelhano, & J. M. Henderson, 2006). Previous studies utilized stimuli of restricted size, a fixed head position, and context defined by the global spatial configuration of the scene. Thus, it is unknown whether similar effects generalize to natural viewing environments and to context defined by local object co-occurrence. Here, with a mobile eye tracker, we investigated the effects of object co-occurrence on search performance under naturalistic conditions. Observers searched for low-visibility target objects on tables cluttered with everyday objects. Targets were either located adjacent to larger, more visible "cue" objects that they regularly co-occurred in natural scenes (expected condition) or elsewhere in the display, surrounded by unrelated objects (unexpected condition). Mean search times were shorter for targets at expected locations as compared to unexpected locations. Additionally, context guided eye movements, as more fixations were directed toward cue objects than other non-target objects, particularly when the cue was contextually relevant to the current search target. These results could not be accounted for by image saliency models. Thus, we conclude that object co-occurrence can serve as a contextual cue to facilitate search and guide eye movements in natural environments.     

Attenuation of perceived motion smear during vergence and pursuit tracking

When the eyes move, the images of stationary objects sweep across the retina. Despite this motion of the retinal image and the substantial integration of visual signals across time, physically stationary objects typically do not appear to be smeared during eye movements. Previous studies indicated that the extent of perceived motion smear is smaller when a stationary target is presented during pursuit or saccadic eye movements than when comparable motion of the retinal image occurs during steady fixation. In this study, we compared the extent of perceived motion smear for a stationary target during smooth pursuit and vergence eye movements with that for a physically moving target during fixation. For a target duration of 100 ms or longer, perceived motion smear is substantially less when the motion of the retinal image results from vergence or pursuit eye movements than when it results from the motion of a target during fixation. The reduced extent of perceived motion smear during eye movements compared to fixation cannot be accounted for by different spatio-temporal interactions between visual targets or by unequal attention to the moving test spot under these two types of conditions. We attribute the highly similar attenuation of perceived smear during vergence and pursuit to a comparable action of the extra-retinal signals for disjunctive and conjugate eye movements.     

Saccadic omission: Why we do not see a grey-out during a saccadic eye movement

We investigated why we have no perception of a smeared image resulting from the reduction in contrast (grey-out) occurring at the retina during saccadic eye movements. By turning on light in the experimental room only during the eye movement, we were able to show that this grey-out was perceived as a smeared image of the visual scene. However, when the experimental room was illuminated before and/or after the saccade as well as during the saccade, perception of the grey-out was obliterated. During a period of fixation, perception of a blank image comparable in duration to an eye movement could also be eliminated by a preceding or following clear image. We conclude that lack of perception during saccadic eye movements made in normal contoured environments results primarily from the visual “masking” effect of a clear image before and/or after the eye movement acting on the grey-out during the eye movement. This “saccadic omission” is entirely a visual phenomenon and is far more powerful than the usually studied elevation of visual threshold for detection of a flash, “saccadic suppression.”     

The potency of people in pictures: evidence from sequences of eye fixations

Does the presence of people in a natural scene affect the way that we inspect that picture? Previous research suggests that we have a natural tendency to look at the social information before other items in a scene. There is also evidence that accuracy of visual memory and the way we move our eyes are related. This experiment investigated whether eye movements differed when participants correctly and incorrectly identified stimuli at recognition, and how this is affected by the presence of people. Eye movements were recorded from 15 participants while they inspected photographs at encoding and during a recognition memory test. Half of the pictures contained people and half did not. The presence of people increased recognition accuracy and affected average fixation duration and average saccadic amplitude. Accuracy was not affected by the size of the Region of Interest (RoI), the number of people in the picture, or the distance of the person from the center. Analyses of the order and pattern of fixations showed a high similarity between encoding and recognition in all conditions, but the lack of relationship between string similarity and recognition accuracy challenges the idea that the reproduction of eye movements alone is enough to create a memory advantage.     

Neuronal mechanisms of visual stability

Human vision is stable and continuous in spite of the incessant interruptions produced by saccadic eye movements. These rapid eye movements serve vision by directing the high resolution fovea rapidly from one part of the visual scene to another. They should detract from vision because they generate two major problems: displacement of the retinal image with each saccade and blurring of the image during the saccade. This review considers the substantial advances in understanding the neuronal mechanisms underlying this visual stability derived primarily from neuronal recording and inactivation studies in the monkey, an excellent model for systems in the human brain. For the first problem, saccadic displacement, two neuronal candidates are salient. First are the neurons in frontal and parietal cortex with shifting receptive fields that provide anticipatory activity with each saccade and are driven by a corollary discharge. These could provide the mechanism for a retinotopic hypothesis of visual stability and possibly for a transsaccadic memory hypothesis, The second neuronal mechanism is provided by neurons whose visual response is modulated by eye position (gain field neurons) or are largely independent of eye position (real position neurons), and these neurons could provide the basis for a spatiotopic hypothesis. For the second problem, saccadic suppression, visual masking and corollary discharge are well established mechanisms, and possible neuronal correlates have been identified for each.     

Spatiotopic and retinotopic memory in the context of natural images

Neural responses throughout the visual cortex encode stimulus location in a retinotopic (i.e., eye-centered) reference frame, and memory for stimulus position is most precise in retinal coordinates. Yet visual perception is spatiotopic: objects are perceived as stationary, even though eye movements cause frequent displacement of their location on the retina. Previous studies found that, after a single saccade, memory of retinotopic locations is more accurate than memory of spatiotopic locations. However, it is not known whether various aspects of natural viewing affect the retinotopic reference frame advantage. We found that the retinotopic advantage may in part depend on a retinal afterimage, which can be effectively nullified through backwards masking. Moreover, in the presence of natural scenes, spatiotopic memory is more accurate than retinotopic memory, but only when subjects are provided sufficient time to process the scene before the eye movement. Our results demonstrate that retinotopic memory is not always more accurate than spatiotopic memory and that the fidelity of memory traces in both reference frames are sensitive to the presence of contextual cues.     

Low-level visual saliency does not predict change detection in natural scenes

Saliency models of eye guidance during scene perception suggest that attention is drawn to visually conspicuous areas having high visual salience. Despite such low-level visual processes controlling the allocation of attention, higher level information gained from scene knowledge may also control eye movements. This is supported by the findings of eye-tracking studies demonstrating that scene-inconsistent objects are often fixated earlier than their consistent counterparts. Using a change blindness paradigm, changes were made to objects that were either consistent or inconsistent with the scene and that had been measured as having high or low visual salience (according to objective measurements). Results showed that change detection speed and accuracy for objects with high visual salience did not differ from those having low visual salience. However, changes in scene-inconsistent objects were detected faster and with higher accuracy than those in scene-consistent objects for both high and low visually salient objects. We conclude that the scene-inconsistent change detection advantage is a true top-down effect and is not confounded by low-level visual factors and may indeed override such factors when viewing complex naturalistic scenes.     

Visual working memory for briefly presented scenes

Previous studies have painted a conflicting picture on the amount of visual information humans can extract from viewing a natural scene briefly. Although some studies suggest that a single glimpse is sufficient to put about five visual objects in memory, others find that not much is retained in visual memory even after prolonged viewing. Here we tested subjects' visual working memory (VWM) for a briefly viewed scene image. A sample scene was presented for 250 ms and masked, followed 1000 ms later by a comparison display. We found that subjects remembered fewer than one sample object. Increasing the viewing duration to about 15 s significantly enhanced performance, with approximately five visual objects remembered. We suggest that adequate encoding of a scene into VWM requires a long duration, and that visual details can accumulate in memory provided that the viewing duration is sufficiently long.     

Attention during sequences of saccades along marked and memorized paths

Natural scenes are explored by combinations of saccadic eye movements and shifts of attention. The mechanisms that coordinate attention and saccades during ordinary viewing are not well understood because studies linking saccades and attention have focused mainly on single saccades made in isolation. This study used an orientation discrimination task to examine attention during sequences of saccades made through an array of targets and distractors. Perceptual measures showed that attention was distributed along saccadic paths when the paths were marked by color cues. When paths were followed from memory, attention rarely spread beyond the goal of the upcoming saccade. These different distributions of attention suggest the involvement of separate processes of attentional control during saccadic planning, one triggered by top-down selection of the saccadic target, and the other by activation linked to visual mechanisms not tied directly to saccadic planning. The concurrent activity of both processes extends the effective attentional field without compromising the accuracy, precision, or timing of saccades.     

Motor signals in visual localization

We demonstrate a strong sensory-motor coupling in visual localization in which experimental modification of the control of saccadic eye movements leads to an associated change in the perceived location of objects. Amplitudes of saccades to peripheral targets were altered by saccadic adaptation, induced by an artificial step of the saccade target during the eye movement, which leads the oculomotor system to recalibrate saccade parameters. Increasing saccade amplitudes induced concurrent shifts in perceived location of visual objects. The magnitude of perceptual shift depended on the size and persistence of errors between intended and actual saccade amplitudes. This tight agreement between the change of eye movement control and the change of localization shows that perceptual space is shaped by motor knowledge rather than simply constructed from visual input.     

Eye movements during visual search in patients with glaucoma

ABSTRACT: BACKGROUND: Glaucoma has been shown to lead to disability in many daily tasks including visual search. This study aims to determine whether the saccadic eye movements of people with glaucoma differ from those of people with normal vision, and to investigate the association between eye movements and impaired visual search. METHODS: Forty patients (mean age: 67 [SD: 9] years) with a range of glaucomatous visual field (VF) defects in both eyes (mean best eye mean deviation [MD]: --5.9 (SD: 5.4) dB) and 40 age-related people with normal vision (mean age: 66 [SD: 10] years) were timed as they searched for a series of target objects in computer displayed photographs of real world scenes. Eye movements were simultaneously recorded using an eye tracker. Average number of saccades per second, average saccade amplitude and average search duration across trials were recorded. These response variables were compared with measurements of VF and contrast sensitivity. RESULTS: The average rate of saccades made by the patient group was significantly smaller than the number made by controls during the visual search task (P = 0.02; mean reduction of 5.6 % (95 % CI: 0.1 to 10.4 %). There was no difference in average saccade amplitude between the patients and the controls (P = 0.09). Average number of saccades was weakly correlated with aspects of visual function, with patients with worse contrast sensitivity (PR logCS; Spearman's rho: 0.42; P = 0.006) and more severe VF defects (best eye MD; Spearman's rho: 0.34; P = 0.037) tending to make less eye movements during the task. Average detection time in the search task was associated with the average rate of saccades in the patient group (Spearman's rho = [MINUS SIGN]0.65; P < 0.001) but this was not apparent in the controls. CONCLUSIONS: The average rate of saccades made during visual search by this group of patients was fewer than those made by people with normal vision of a similar average age. There was wide variability in saccade rate in the patients but there was an association between an increase in this measure and better performance in the search task. Assessment of eye movements in individuals with glaucoma might provide insight into the functional deficits of the disease.     

Eye movements during rapid pointing under risk

We recorded saccadic eye movements during visually-guided rapid pointing movements under risk. We intended to determine whether saccadic end points are necessarily tied to the goals of rapid pointing movements or whether, when the visual features of a display and the goals of a pointing movement are different, saccades are driven by low-level features of the visual stimulus. Subjects pointed at a stimulus configuration consisting of a target region and a penalty region. Each target hit yielded a gain of points; each penalty hit incurred a loss of points. Late responses were penalized. The luminance of either target or penalty region was indicated by a disk which differed significantly from the background in luminance, while the other region was indicated by a thin circle. In subsequent experiments, we varied the visual salience of the stimulus configuration and found that manual responses followed near-optimal strategies maximizing expected gain, independent of the salience of the target region. We suggest that the final eye position is partially pre-programmed prior to hand movement initiation. While we found that manipulations of the visual salience of the display determined the end point of the initial saccade we also found that subsequent saccades are driven by the goal of the hand movement.     

Task demands and binocular eye movements

Humans make rapid movements of their eyes several times a second that enable them to examine objects located at different positions in space with both of their eyes. Much of our understanding of these binocular movements comes from studies using experienced observers performing repetitive, unnatural tasks. But what eye movements are made when na?ve observers perform tasks demanding specific binocular visual information? We examined the binocular eye movements produced by observers performing two tasks differing in the visual information needed for their completion. Our motivation for doing this was to examine the role and function of binocular eye movements when making decisions. We considered the fixation strategies adopted by observers, the effects of the task on the dynamics of saccadic eye movements, and the combination of vergence and version in gaze shifts. We report that the task-dependent use of visual information can have a strong influence on the patterns of fixations, whilst not influencing saccade dynamics. Our data provide some support for the notion that observers choose and fixate a notional reference point in the scene when making judgments about depth structure.     

Competition and selection during visual processing of natural scenes and objects

When a visual scene, containing many discrete objects, is presented to our retinae, only a subset of these objects will be explicitly represented in visual awareness. The number of objects accessing short-term visual memory might be even smaller. Finally, it is not known to what extent "ignored" objects (those that do not enter visual awareness) will be processed--or recognized. By combining free recall, forced-choice recognition and visual priming paradigms for the same natural visual scenes and subjects, we were able to estimate these numbers, and provide insights as to the fate of objects that are not explicitly recognized in a single fixation. When presented for 250 ms with a scene containing 10 distinct objects, human observers can remember up to 4 objects with full confidence, and between 2 and 3 more when forced to guess. Importantly, the objects that the subjects consistently failed to report elicited a significant negative priming effect when presented in a subsequent task, suggesting that their identity was represented in high-level cortical areas of the visual system, before the corresponding neural activity was suppressed during attentional selection. These results shed light on neural mechanisms of attentional competition, and representational capacity at different levels of the human visual system.     

Eye movements and attention: The role of pre-saccadic shifts of attention in perception,memory and the control of saccades

Saccadic eye movements and perceptual attention work in a coordinated fashion to allow selection of the objects, features or regions with the greatest momentary need for limited visual processing resources. This study investigates perceptual characteristics of pre-saccadic shifts of attention during a sequence of saccades using the visual manipulations employed to study mechanisms of attention during maintained fixation. The first part of this paper reviews studies of the connections between saccades and attention, and their significance for both saccadic control and perception. The second part presents three experiments that examine the effects of pre-saccadic shifts of attention on vision during sequences of saccades. Perceptual enhancements at the saccadic goal location relative to non-goal locations were found across a range of stimulus contrasts, with either perceptual discrimination or detection tasks, with either single or multiple perceptual targets, and regardless of the presence of external noise. The results show that the preparation of saccades can evoke a variety of attentional effects, including attentionally-mediated changes in the strength of perceptual representations, selection of targets for encoding in visual memory, exclusion of external noise, or changes in the levels of internal visual noise. The visual changes evoked by saccadic planning make it possible for the visual system to effectively use saccadic eye movements to explore the visual environment.     

Scene and screen center bias early eye movements in scene viewing

In laboratory studies of visual perception, images of natural scenes are routinely presented on a computer screen. Under these conditions, observers look at the center of scenes first, which might reflect an advantageous viewing position for extracting visual information. This study examined an alternative possibility, namely that initial eye movements are drawn towards the center of the screen. Observers searched visual scenes in a person detection task, while the scenes were aligned with the screen center or offset horizontally (Experiment 1). Two central viewing effects were observed, reflecting early visual biases to the scene and the screen center. The scene effect was modified by person content but is not specific to person detection tasks, while the screen bias cannot be explained by the low-level salience of a computer display (Experiment 2). These findings support the notion of a central viewing tendency in scene analysis, but also demonstrate a bias to the screen center that forms a potential artifact in visual perception experiments.     

Limited flexibility in the filter underlying saccadic targeting

The choice of where to look in a visual scene depends on visual processing of information from potential target locations. We examined to what extent the sampling window, or filter, underlying saccadic eye movements is under flexible control and adjusted to the behavioural task demands. Observers performed a contrast discrimination task with systematic variations in the spatial scale and location of the visual signals: small (sigma=0.175 degrees ) or large (sigma=0.8 degrees ) Gaussian signals were presented 4.5 degrees , 6 degrees , or 9 degrees away from central fixation. In experiment 1, we measured the accuracy of the first saccade as a function of target contrast. The efficiency of saccadic targeting decreased with increases in both scale and eccentricity. In experiment 2, the filter underlying saccadic targeting was estimated with the classification image method. We found that the filter (1) had a center-surround organisation, even though the signal was Gaussian; (2) was much too small for the large scale items; (3) remained constant up to the largest measured eccentricity of 9 degrees . The filter underlying the decision of where to look is not fixed, and can be adjusted to the task demands. However, there are clear limits to this flexibility. These limits reflect the coding of visual information by early mechanisms, and the extent to which the neural circuitry involved in programming saccadic eye movements is able to appropriately weigh and combine the outputs from these mechanisms.