Artificial vision support system (AVS2) for improved prosthetic vision

State-of-the-art and upcoming camera-driven, implanted artificial vision systems provide only tens to hundreds of electrodes, affording only limited visual perception for blind subjects. Therefore, real time image processing is crucial to enhance and optimize this limited perception. Since tens or hundreds of pixels/electrodes allow only for a very crude approximation of the typically megapixel optical resolution of the external camera image feed, the preservation and enhancement of contrast differences and transitions, such as edges, are especially important compared to picture details such as object texture. An Artificial Vision Support System (AVS²) is devised that displays the captured video stream in a pixelation conforming to the dimension of the epi-retinal implant electrode array. AVS², using efficient image processing modules, modifies the captured video stream in real time, enhancing ‘present but hidden’ objects to overcome inadequacies or extremes in the camera imagery. As a result, visual prosthesis carriers may now be able to discern such objects in their ‘field-of-view’, thus enabling mobility in environments that would otherwise be too hazardous to navigate. The image processing modules can be engaged repeatedly in a user-defined order, which is a unique capability. AVS² is directly applicable to any artificial vision system that is based on an imaging modality (video, infrared, sound, ultrasound, microwave, radar, etc.) as the first step in the stimulation/processing cascade, such as: retinal implants (i.e. epi-retinal, sub-retinal, suprachoroidal), optic nerve implants, cortical implants, electric tongue stimulators, or tactile stimulators.     

Artificial organs: recent progress in artificial hearing and vision

Artificial sensory organs are a prosthetic means of sending visual or auditory information to the brain by electrical stimulation of the optic or auditory nerves to assist visually impaired or hearing-impaired people. However, clinical application of artificial sensory organs, except for cochlear implants, is still a trial-and-error process. This is because how and where the information transmitted to the brain is processed is still unknown, and also because changes in brain function (plasticity) remain unknown, even though brain plasticity plays an important role in meaningful interpretation of new sensory stimuli. This article discusses some basic unresolved issues and potential solutions in the development of artificial sensory organs such as cochlear implants, brainstem implants, artificial vision, and artificial retinas. This article is a translation of an article that first appeared in Japanese in Jinkozoki 2007;36(3):198–200     

基于人工视网膜神经节细胞感受野的图像边缘检测

用光电池组装一个工感受野，在一维上模拟视网膜神经节细胞的零交叉滤波特性，进一步用该工工感受野构建一个图像边缘检测系统，并成功地用系统检测到图像边缘。一与般的基于视觉原理的图像系统相比，这种以人工感受野灵基础的系统，在图像输入的同时即可并行地提取图像边缘信息，而更接近于动物视觉系统原型，此结果将为人工视觉和图像技术的发展提供新的探索。     

Simulation of a phosphene-based visual field: Visual acuity in a pixelized vision system

A visual prosthesis for the blind using electrical stimulation of the visual cortex will require the development of an array of electrodes. Passage of current through these electrodes is expected to create a visual image made up of a matrix of discrete phosphenes. The quality of the visual sense thus provided will be a function of many parameters, particularly the number of electrodes and their spacing. We are conducting a series of psychophysical experiments with a portable “phosphene” simulator to obtain estimates of suitable values for electrode number and spacing. The simulator consists of a small video camera and monitor worn by a normally sighted human subject. To simulate a discrete phosphene field, the monitor is masked by an opaque perforated film. The visual angle subtended by images from the masked monitor is 1.7° or less, depending on the mask, and falls within the fovea of the subject. In the study presented here, we measured visual acuity as a function of the number of pixels and their spacing in the mask. Visual acuity was inversely proportional to pixel density, and trained subjects could achieve about 20/26 visual acuity with a 1024 pixel image. We conclude that 625 electrodes implanted in a 1 cm by 1 cm area near the foveal representation of the visual cortex should produce a phosphene image with a visual acuity of approximately 20/30. Such an acuity could provide useful restoration of functional vision for the profoundly blind.     

Image processing strategies based on saliency segmentation for object recognition under simulated prosthetic vision

Background and objectiveCurrent retinal prostheses can only generate low-resolution visual percepts constituted of limited phosphenes which are elicited by an electrode array and with uncontrollable color and restricted grayscale. Under this visual perception, prosthetic recipients can just complete some simple visual tasks, but more complex tasks like face identification/object recognition are extremely difficult. Therefore, it is necessary to investigate and apply image processing strategies for optimizing the visual perception of the recipients. This study focuses on recognition of the object of interest employing simulated prosthetic vision.MethodWe used a saliency segmentation method based on a biologically plausible graph-based visual saliency model and a grabCut-based self-adaptive-iterative optimization framework to automatically extract foreground objects. Based on this, two image processing strategies, Addition of Separate Pixelization and Background Pixel Shrink, were further utilized to enhance the extracted foreground objects.Resultsi) The results showed by verification of psychophysical experiments that under simulated prosthetic vision, both strategies had marked advantages over Direct Pixelization in terms of recognition accuracy and efficiency. ii) We also found that recognition performance under two strategies was tied to the segmentation results and was affected positively by the paired-interrelated objects in the scene.ConclusionThe use of the saliency segmentation method and image processing strategies can automatically extract and enhance foreground objects, and significantly improve object recognition performance towards recipients implanted a high-density implant.     

基于图像显著性的人工视觉图像处理策略

目的:在假体设备中的视觉信息处理模块引入适当的图像处理策略,优化低分辨率下的人工视觉信息,是解决假体植入者获取的视觉信息有限问题的一种可行方法。方法:基于全局亮度对比度特征的图像显著性检测算法（LC）,结合颜色空间变换和视觉注意力仿真处理模型,提出一种面向视网膜假体人工视觉信息处理策略。通过两个标准图像测试数据集对图像处理结果进行评估,同时对流行病学调查结果选取的盲人常用物体图片进行仿真。结果:评估结果验证了在提取图像前景方面与原始LC算法对比的优越性,仿真结果验证了在人工视觉条件下应用的可行性。结论:本文提出的策略有助于视网膜假体植入者在日常生活场景中更好地完成物体识别等基本视觉任务。     

Stereo vision enhances the learning of a catching skill

The aim of this study was to investigate the contribution of stereo vision to the acquisition of a natural interception task. Poor catchers with good (N = 8; Stereo+) and weak (N = 6; Stereo-) stereo vision participated in an intensive training program spread over 2 weeks, during which they caught over 1,400 tennis balls in a pre-post-retention design. While the Stereo+ group improved from a catching percentage of 18% to 59%, catchers in the Stereo- group did not significantly improve (from 10 to 31%), this progress being indifferent from a control group (N = 9) that did not practice at all. These results indicate that the development and use of of compensatory cues for depth perception in people with weak stereopsis is insufficient to successfully deal with interceptions under high temporal constraints, and that this disadvantage cannot be fully attenuated by specific and intensive training.     

Monocular vision and increased distance reducing the effects of visual manipulation on body sway

The goal of this study was to examine the coupling between visual information and body sway with binocular and monocular vision at two distances from the front wall of a moving room. Ten participants stood as still as possible inside of a moving room facing the front wall in conditions that combined room movement with monocular/binocular vision and distance from the front wall (75 and 150 cm). Visual information effect on body sway decreased with monocular vision and with increased distance from the front wall. In addition, the combination of monocular vision with the farther distance resulted in the smallest body sway response to the driving stimulus provided by the moving room. These results suggest that binocular vision near the front wall provides visual information of a better quality than the monocular vision far from the front wall. We discuss the results with respect to two modes of visual detection of body sway: ocular and extraocular.     

Binocular single vision and depth discrimination. Receptive field disparities for central and peripheral vision and binocular interaction on peripheral single units in cat striate cortex

Summary Of binocularly-activated striate neurons only a proportion have their two receptive fields in exactly corresponding positions in the contralateral hemifield. Those which are not corresponding are said to show receptive field disparity. Because the eyes diverge in the anaesthetized and paralyzed preparation, the binocular receptive fields are horizontally separate. With increasing retinal eccentricity there is a gradual decrease in this horizontal separation as well as progressive changes in the local receptive field disparities. With increasing horizontal retinal eccentricity there is a progressive increase in horizontal receptive field disparities together with a smaller decrease in vertical disparities. Receptive field disparities are relatively unaffected by increasing vertical retinal eccentricity. A neurophysiological theory for binocular single vision and depth discrimination is put forward as a theoretical framework for the construction of the horopter for the cat as well as a region analogous to Panum's fusional area in man. Observations have been made on the responses, particularly to moving slit stimuli, of units with peripherally-located receptive fields. For several binocular units it was possible to study the full range of the binocular interaction when the two receptive fields were moved from exact correspondence to positions of increasing non-alignment.     

Adaptation of ocular vergence to stimulation with large disparities

Summary Ocular vergence movements were measured with a scleral coil technique under stabilized viewing conditions for disparity. Crossed disparity steps, ranging between 0.25 and 10 deg, of three different targets were imposed. Ocular vergence responses consisted of converging movements with an initially constant velocity. This velocity increased with the magnitude of disparity up to about 4 deg and decreased for larger disparities. For disparities up to 2 deg the responses saturated at the limit of convergence. For larger disparities responses were transient, i.e. after large converging movements the angle of convergence gradually declined to about its initial value. For disparities larger than 5 deg amplitudes of the transient responses decreased and occasionally responses were completely absent. The transient character of responses was apparently due to adaptation of the vergence system to a specific disparity, since responses to different disparities could still be induced. Probing of the vergence system with two successive disparity steps of different magnitudes showed that adaptation was selective for a limited range of disparities around the adapting disparity stimulus. Stabilized disparity pulses with durations ranging from 100 to 800 ms induced ocular vergence movements following the time integral of disparity rather than momentary disparity. This indicated that the part of the vergence system sensitive to disparity has mainly integrative properties for large as well as small disparities. Stimulation with trains of shortlasting disparity pulses showed that adaptation also occurred under normal viewing conditions. They further revealed that adaptation is erased during a short period after each blink.     

Differential effects of non-informative vision and visual interference on haptic spatial processing

The primary purpose of this study was to examine the effects of non-informative vision and visual interference upon haptic spatial processing, which supposedly derives from an interaction between an allocentric and egocentric reference frame. To this end, a haptic parallelity task served as baseline to determine the participant-dependent biasing influence of the egocentric reference frame. As expected, large systematic participant-dependent deviations from veridicality were observed. In the second experiment we probed the effect of non-informative vision on the egocentric bias. Moreover, orienting mechanisms (gazing directions) were studied with respect to the presentation of haptic information in a specific hemispace. Non-informative vision proved to have a beneficial effect on haptic spatial processing. No effect of gazing direction or hemispace was observed. In the third experiment we investigated the effect of simultaneously presented interfering visual information on the haptic bias. Interfering visual information parametrically influenced haptic performance. The interplay of reference frames that subserves haptic spatial processing was found to be related to both the effects of non-informative vision and visual interference. These results suggest that spatial representations are influenced by direct cross-modal interactions; inter-participant differences in the haptic modality resulted in differential effects of the visual modality.     

A model retinal interface based on directed neuronal growth for single cell stimulation

In this work, we use cell micropatterning technologies to direct neuronal growth to individual electrodes, and demonstrate that such an approach can achieve selective stimulation and lower stimulation thresholds than current field-effect based retinal prostheses. Rat retinal ganglion cells (RGCs) were purified through immunopanning techniques, and microcontact printing (μCP) was applied to align and pattern laminin on a microelectrode array, on which the RGCs were seeded and extended neurites along the pattern to individual electrodes. The stimulation threshold currents of RGCs micropatterned to electrodes were found to be significantly less than those of non-patterned RGCs over a wide range of electrode-soma distances, as determined with calcium imaging techniques. Moreover, the stimulation threshold for micropatterned cells was found to be independent of electrode-soma distance, and there was no significant effect of μCP on cell excitability. The effects of additional stimulation parameters, such as electrode size and pulse duration, on threshold currents were determined. The stimulation results quantitatively demonstrate the potential benefits of a retinal prosthetic interface based on directed neuronal growth.     

Neuronal expression of c-Fos after epicortical and intracortical electric stimulation of the primary visual cortex

Electrical stimulation of the primary visual cortex (V1) is an experimental approach for visual prostheses. We here compared the response to intracortical and epicortical stimulation of the primary visual cortex by using c-Fos immunoreactivity as a marker for neuronal activation.The primary visual cortex of male Sprague Dawley rats was unilaterally stimulated for four hours using bipolar electrodes placed either intracortically in layer IV (n = 26) or epicortically (n = 20). Four different current intensities with a constant pulse width of 200 μs and a constant frequency of 10 Hz were used, for intracortical stimulation with an intensity of 0 μA (sham-stimulation), 10 μA, 20 μA and 40 μA, and for epicortical stimulation 0 μA, 400 μA, 600 μA and 800 μA. Subsequently all animals underwent c-Fos immunostaining and c-Fos expression was assessed in layer I–VI of the primary visual cortex within 200 μm and 400 μm distance to the stimulation site. C-Fos expression was higher after intracortical stimulation compared to epicortical stimulation, even though ten times lower current intensities were applied. Furthermore intracortical stimulation resulted in more focal neuronal activation than epicortical stimulation. C-Fos expression was highest after intracortical stimulation with 20 μA compared to all other intensities. Epicortical stimulation showed a linear increase of c-Fos expression with the highest expression at 800 μA. Sham stimulation showed similar expression of c-Fos in both hemispheres. The contralateral hemisphere was not affected by intracortical or epicortical stimulation of either intensities. In summary, intracortical stimulation resulted in more focal neuronal activation with less current than epicortical stimulation. This model may be used as a simple but reliable model to evaluate electrodes for microstimulation of the primary visual cortex before testing in more complex settings.     

Simulation of intra-orbital optic nerve electrical stimulation

In blind subjects who still have functional retinal ganglion cells, electrical stimuli applied to the optic nerve can produce localised visual sensations. This has been demonstrated with an intracranially implanted self-sizing spiral cuff electrode, but, to avoid skull opening, intra-orbital cuff implantation is now considered. In its orbital segment, the optic nerve is surrounded by subarachnoidal cerebrospinal fluid (CSF) and dura mater. Dura mater is a tough fibrous tissue that can impede electrical stimulation. In the study, the issue of whether or not to remove the dura mater at the implantation site was addressed using simulation on numerical models. Several volume conductor models were built representing, respectively: the cuff implanted directly around the nerve; the cuff over the nerve after connective tissue encapsulated the implant; and the cuff electrode placed around the dura mater. Stimulation-induced electric potential fields were computed for these configurations using a full 3D finite elements software. Responses of fibres within the nerve were computed. A large range of dural conductivities and several CSF thicknesses were considered. In all simulated conditions, the presence of dura mater around a layer of CSF increased excitation thresholds. Selectivity performance also decreased, but was found to be independent of the CSF thickness. However, simulations showed that, if the diameter of the cuff electrode is adapted to the target nerve, the injected charge associated with activation is limited within a reasonable range. Electrical stimulation of the optic nerve with a cuff electrode implanted around the dura mater should therefore be feasible.     

Enhanced responsiveness of human extravisual areas to photic stimulation in patients with severely reduced vision

Lesions in the primary visual cortex induce severe loss of visual perception. Depending on the size of the lesion, the visual field might be affected by small scotomas, hemianopia, or complete loss of vision (cortical blindness). In many cases, the whole visual field of the patient is affected by the lesion, but diffuse light-dark discrimination remains (residual rudimentary vision, RRV). In other cases, a sparing of a few degrees can be found (severely reduced vision, SRV).In a follow-up study, we mapped visually induced cerebral activation of three subjects with SRV using functional magnetic resonance imaging. We were especially interested in the visual areas that would be activated if subjects could perceive the stimulus consciously although information flow from V1 to higher visual areas was strongly reduced or virtually absent. Because subjects were only able to discriminate strong light from darkness, we used goggles flashing intense red light at a frequency of 3 Hz for full visual field stimulation. Besides reduced activation in V1, we found activation in the parietal cortex, the frontal eye fields (FEF), and the supplementary eye fields (SEF). In all patients, FEF activation was pronounced in the right hemisphere. These patterns were never seen in healthy volunteers. In a patient who recovered completely, we observed that extrastriate activation disappeared in parallel with the visual field restitution. This result suggests that damage to the primary visual cortex changes the responsiveness of parietal and extravisual frontal areas in patients with SRV. This unexpected result might be explained by increased stimulus-related activation of attention-related networks.     

Retinal Prosthesis： A Potential Benefit for the Blind

Retinal degenerative diseases may induce the degeneration of outer retina and in turn, blindness. Nevertheless, due to the maintenance of inner retina, the coding and processing of visual neurons responses are still able to be executed naturally. Therefore, an effective retinal prosthesis device may be developed by mimicking the function of outer retina： transferring the visual light into artificial stimulus and delivering the stimulus to the retina aiming to evoke the neural responses. As two main developing directions for current retinal prosthesis,epiretinal （ER） and subretinal （SR） prosthesis are both undergoing experimental stage and possessing advantages and limitations. Further investigations in power supply, biocompatibility, etc. are still required. Additionally, suprachoroidal transretinal stimulation （STS） and neurotransmitter-induced stimulation as some other alternatives in retinal prosthesis are also considered as promising research directions, although they are not mature enough to be applied commercially, either.