Neocognitron trained with winner-kill-loser rule

The neocognitron, which was proposed by Fukushima (1980), is a hierarchical multi-layered neural network capable of robust visual pattern recognition. It acquires the ability to recognize patterns through learning.This paper proposes a new rule for competitive learning, named winner-kill-loser, and apply it to the neocognitron. The winner-kill-loser rule resembles the winner-take-all rule. Every time when a training stimulus is presented, non-silent cells compete with each other. The winner, however, not only takes all, but also kills losers. In other words, the winner learns the training stimulus, and losers are removed from the network. If all cells are silent, a new cell is generated and it learns the training stimulus. Thus feature-extracting cells gradually come to distribute uniformly in the feature space. The use of winner-kill-loser rule is not limited to the neocognitron. It is useful for various types of competitive learning, in general.This paper also proposes several improvements made on the neocognitron: such as, disinhibition to the inhibitory surround in the connections to C-cells (or complex cells) from S-cells (or simple cells); and square root shaped saturation in the input-to-output characteristics of C-cells. As a result of these improvements, the recognition rate of the neocognitron has been largely increased.     

Artificial vision by multi-layered neural networks: Neocognitron and its advances

The neocognitron is a neural network model proposed by Fukushima (1980). Its architecture was suggested by neurophysiological findings on the visual systems of mammals. It is a hierarchical multi-layered network. It acquires the ability to robustly recognize visual patterns through learning. Although the neocognitron has a long history, modifications of the network to improve its performance are still going on. For example, a recent neocognitron uses a new learning rule, named add-if-silent, which makes the learning process much simpler and more stable. Nevertheless, a high recognition rate can be kept with a smaller scale of the network. Referring to the history of the neocognitron, this paper discusses recent advances in the neocognitron. We also show that various new functions can be realized by, for example, introducing top-down connections to the neocognitron: mechanism of selective attention, recognition and completion of partly occluded patterns, restoring occluded contours, and so on.     

Increasing robustness against background noise: Visual pattern recognition by a neocognitron

The neocognitron is a hierarchical multi-layered neural network capable of robust visual pattern recognition. It has been demonstrated that recent versions of the neocognitron exhibit excellent performance for recognizing handwritten digits. When characters are written on a noisy background, however, recognition rate was not always satisfactory. To find out the causes of vulnerability to noise, this paper analyzes the behavior of feature-extracting S-cells. It then proposes the use of subtractive inhibition to S-cells from V-cells, which calculate the average of input signals to the S-cells with a root-mean-square. Together with this, several modifications have also been applied to the neocognitron. Computer simulation shows that the new neocognitron is much more robust against background noise than the conventional ones.     

Analysis of the process of visual pattern recognition by the neocognitron

A neural network model of visual pattern recognition called the “neocognitron,” was earlier proposed by the author. It is capable of deformation-invariant visual pattern recognition. After learning, it can recognize input patterns without being affected by deformation, changes in size, or shifts in position. This paper offers a mathematical analysis of the process of visual pattern recognition by the neocognitron. The neocognitron is a hierarchical multilayered network. Its initial stage is an input layer, and each succeeding stage has a layer of “S-cells” followed by a layer of “C-cells.” Thus, in the whole network, layers of S-cells and C-cells are arranged alternately. The process of feature extraction by an S-cell is analyzed mathematically in this paper, and the role of the C-cells in deformation-invariant pattern recognition is discussed.     

Neocognitron capable of incremental learning.

This paper proposes a new neocognitron that accepts incremental learning, without giving a severe damage to old memories or reducing learning speed. The new neocognitron uses a competitive learning, and the learning of all stages of the hierarchical network progresses simultaneously. To increase the learning speed, conventional neocognitrons of recent versions sacrificed the ability of incremental learning, and used a technique of sequential construction of layers, by which the learning of a layer started after the learning of the preceding layers had completely finished. If the learning speed is simply set high for the conventional neocognitron, simultaneous construction of layers produces many garbage cells, which become always silent after having finished the learning. The proposed neocognitron with a new learning method can prevent the generation of such garbage cells even with a high learning speed, allowing incremental learning.     

Self-organization of shift-invariant receptive fields

This paper proposes a new learning rule by which cells with shift-invariant receptive fields are self-organized. With this learning rule, cells similar to simple and complex cells in the primary visual cortex are generated in a network. To demonstrate the new learning rule, we simulate a three-layered network that consists of an input layer (or the retina), a layer of S-cells (or simple cells), and a layer of C-cells (or complex cells). During the learning, straight lines of various orientations sweep across the input layer. Here both S- and C-cells are created through competition. Although S-cells compete depending on their instantaneous outputs, C-cells compete depending on the traces (or temporal averages) of their outputs. For the self-organization of S-cells, only winner S-cells increase their input connections in a similar way to that for the neocognitron. In other words, the winner S-cells have LTP (long term potentiation) in their input connections. For the self-organization of C-cells, however, loser C-cells decrease their input connections (LTD=long term depression), while winners increase their input connections (LTP). Here both S- and C-cells are accompanied by inhibitory cells. Modification of inhibitory connections together with excitatory connections is important for creation of C-cells as well as S-cells.     

Neural network model for completing occluded contours

When some parts of a pattern are occluded by other objects, the visual system can often estimate the shape of occluded contours from visible parts of the contours. This paper proposes a neural network model capable of such function, which is called amodal completion. The model is a hierarchical multi-layered network that has bottom-up and top-down signal paths. It contains cells of area V1, which respond selectively to edges of a particular orientation, and cells of area V2, which respond selectively to a particular angle of bend. Using the responses of bend-extracting cells, the model predicts the curvature and location of the occluded contours. Missing contours are gradually extrapolated and interpolated from the visible contours. Computer simulation demonstrates that the model performs amodal completion to various stimuli in a similar way as observed by psychological experiments.     

Control of neuromuscular stimulation for ambulation by complete paraplegics via artificial neural networks

Abstract

The paper describes the application of a neural network (ANN) for controlling a functional neuromuscular stimulation (FNS) system to facilitate patient-responsive ambulation by paralyzed patients with traumatic, thoracic-level spinal cord injuries. The particular ANN that is employed is a modified Adaptive-ResonanceTheory (ART-1) network. It serves as a controller in an FNS system (the Parastep system) that is presently in use by approximately 500 patients worldwide (but still without ANN control) and which was the first and only FNS system approved by FDA. The proposed neural network discriminates above-lesion upper-trunk electromyographic (EMG) time series to activate standing and walking functions under FNS and controls FNS stimuli levels using response-EMG signals. For this particular application, several modifications are introduced into the standard ART-1 ANN. First, a modified on-line learning rule is proposed. The new rule assures bi-directional modification of the stored patterns and prevents noise interference. Second, a new reset rule is proposed, which prevents 'exact matching' when the input is a subset of the chosen pattern. A single ART-1-based structure is being applied to solving two problems, namely (1) signal pattern recognition and limb function determination, and (2) control of stimulation levels. This also facilitates ambulation of paraplegics under FNS, with adequate patient interaction in initial system training, retraining the network when needed, and in allowing patient's manual over-ride in the case of error, where any manual over-ride serves as a re-training input to the neural network. The ANN control facilitates continuous update of control settings during normal use, without formal retraining. [Neurol Res 2001; 23: 472-481]     

A Simple Recurrent Network for Implicit Learning of Temporal Sequences

A behavioural paradigm for learning arbitrary visuo-motor associations established that human observers learn to associate visual objects with their corresponding motor responses faster if the objects follow a temporal rule rather than if they were presented in a random order. Here, we use a simple recurrent network with a back propagation training algorithm adapted to a reinforcement learning scheme. Our simulations fit quantitatively as well as qualitatively to the behavioural results, endorsing the role of temporal context in associative learning scenarios.     

Restoring partly occluded patterns: a neural network model.

This paper proposes a neural network model that has an ability to restore missing portions of partly occluded patterns. It is a multi-layered hierarchical neural network, in which visual information is processed by interaction of bottom-up and top-down signals. Memories of learned patterns are stored in the connections between cells. Occluded parts of a pattern are reconstructed mainly by top-down signals from higher stages of the network, while the unoccluded parts are reproduced mainly by signals from lower stages. The restoration progresses successfully, even if the occluded pattern is a deformed version of a learned pattern. The model tries to complete even an unlearned pattern by interpolating and extrapolating visible edges. Resemblance of local features to other learned patterns are also utilized for the restoration.     

Categorization and decision-making in a neurobiologically plausible spiking network using a STDP-like learning rule

Understanding how the human brain is able to efficiently perceive and understand a visual scene is still a field of ongoing research. Although many studies have focused on the design and optimization of neural networks to solve visual recognition tasks, most of them either lack neurobiologically plausible learning rules or decision-making processes. Here we present a large-scale model of a hierarchical spiking neural network (SNN) that integrates a low-level memory encoding mechanism with a higher-level decision process to perform a visual classification task in real-time. The model consists of Izhikevich neurons and conductance-based synapses for realistic approximation of neuronal dynamics, a spike-timing-dependent plasticity (STDP) synaptic learning rule with additional synaptic dynamics for memory encoding, and an accumulator model for memory retrieval and categorization. The full network, which comprised 71,026 neurons and approximately 133 million synapses, ran in real-time on a single off-the-shelf graphics processing unit (GPU). The network was constructed on a publicly available SNN simulator that supports general-purpose neuromorphic computer chips. The network achieved 92% correct classifications on MNIST in 100 rounds of random sub-sampling, which is comparable to other SNN approaches and provides a conservative and reliable performance metric. Additionally, the model correctly predicted reaction times from psychophysical experiments. Because of the scalability of the approach and its neurobiological fidelity, the current model can be extended to an efficient neuromorphic implementation that supports more generalized object recognition and decision-making architectures found in the brain.     

A Hierarchical Predictive Coding Model of Object Recognition in Natural Images

Predictive coding has been proposed as a model of the hierarchical perceptual inference process performed in the cortex. However, results demonstrating that predictive coding is capable of performing the complex inference required to recognise objects in natural images have not previously been presented. This article proposes a hierarchical neural network based on predictive coding for performing visual object recognition. This network is applied to the tasks of categorising hand-written digits, identifying faces, and locating cars in images of street scenes. It is shown that image recognition can be performed with tolerance to position, illumination, size, partial occlusion, and within-category variation. The current results, therefore, provide the first practical demonstration that predictive coding (at least the particular implementation of predictive coding used here; the PC/BC-DIM algorithm) is capable of performing accurate visual object recognition.     

Recursion in action: An fMRI study on the generation of new hierarchical levels in motor sequences

Generation of hierarchical structures, such as the embedding of subordinate elements into larger structures, is a core feature of human cognition. Processing of hierarchies is thought to rely on lateral prefrontal cortex (PFC). However, the neural underpinnings supporting active generation of new hierarchical levels remain poorly understood. Here, we created a new motor paradigm to isolate this active generative process by means of fMRI. Participants planned and executed identical movement sequences by using different rules: a Recursive hierarchical embedding rule, generating new hierarchical levels; an Iterative rule linearly adding items to existing hierarchical levels, without generating new levels; and a Repetition condition tapping into short term memory, without a transformation rule. We found that planning involving generation of new hierarchical levels (Recursive condition vs. both Iterative and Repetition) activated a bilateral motor imagery network, including cortical and subcortical structures. No evidence was found for lateral PFC involvement in the generation of new hierarchical levels. Activity in basal ganglia persisted through execution of the motor sequences in the contrast Recursive versus Iteration, but also Repetition versus Iteration, suggesting a role of these structures in motor short term memory. These results showed that the motor network is involved in the generation of new hierarchical levels during motor sequence planning, while lateral PFC activity was neither robust nor specific. We hypothesize that lateral PFC might be important to parse hierarchical sequences in a multi‐domain fashion but not to generate new hierarchical levels.     

Interpolating vectors for robust pattern recognition.

This paper proposes a powerful algorithm for pattern recognition, which uses interpolating vectors for classifying patterns. Labeled reference vectors in a multi-dimensional feature space are first produced by a kind of competitive learning. We then assume a situation where virtual vectors, called interpolating vectors, are densely placed along line segments connecting all pairs of reference vectors of the same label. From these interpolating vectors, we choose the one that has the largest similarity to the test vector. Its label shows the result of pattern recognition. In practice, we can get the same result with a simpler process. We applied this method to the neocognitron for handwritten digit recognition and reduced the error rate from 1.52% to 1.02% for a blind test set of 5000 digits.     

A Learner-Independent Knowledge Transfer Approach to Multi-task Learning

This paper proposes a learner-independent multi-task learning (MTL) scheme in which knowledge transfer (KT) is running beyond the learner. In the proposed KT approach, we use minimum enclosing balls (MEBs) as knowledge carriers to extract and transfer knowledge from one task to another. Since the knowledge presented in MEB can be decomposed as raw data, it can be incorporated into any learner as additional training data for a new learning task to improve the learning rate. The effectiveness and robustness of the proposed KT is evaluated, respectively, on multi-task pattern recognition problems derived from synthetic datasets, UCI datasets, and real face image datasets, using classifiers from different disciplines for MTL. The experimental results show that multi-task learners using KT via MEB carriers perform better than learners without-KT, and this has been successfully applied to different classifiers such as k nearest neighbor and support vector machines.     

Divide and Conquer Neural Networks

Determining an effective architecture far a multi-layer feedforward back propagation neural network can be a time-consuming effort. We describe an algorithm called Divide and Conquer Neural Networks (DCN), which creates a feedforward neural network architecture during training, based upon the training examples. The first cell introduced on any layer is trained on all examples. Further cells on a layer are trained primarily on examples not already correctly classified. The learning algorithm is shown to be able to use several different learning rules, including the delta rule and perceptron rule, to modify the link weights one level at a time in the spirit of a perceptron. Error is never propagated backwards through a hidden cell. Examples are shown of networks generated for the exdusive-or, 4 and 5-parity, 2-spirals problem, Iris plant classification, predicting party affiliation from voting records, and the real-valued fuzzy exclusive-or. The results show the algorithm effectively learns viable architectures that can generalize.     

Optimal unsupervised learning in a single-layer linear feedforward neural network

A new approach to unsupervised learning in a single-layer linear feedforward neural network is discussed. An optimality principle is proposed which is based upon preserving maximal information in the output units. An algorithm for unsupervised learning based upon a Hebbian learning rule, which achieves the desired optimality is presented. The algorithm finds the eigenvectors of the input correlation matrix, and it is proven to converge with probability one. An implementation which can train neural networks using only local “synaptic” modification rules is described. It is shown that the algorithm is closely related to algorithms in statistics (Factor Analysis and Principal Components Analysis) and neural networks (Self-supervised Backpropagation, or the “encoder” problem). It thus provides an explanation of certain neural network behavior in terms of classical statistical techniques. Examples of the use of a linear network for solving image coding and texture segmentation problems are presented. Also, it is shown that the algorithm can be used to find “visual receptive fields” which are qualitatively similar to those found in primate retina and visual cortex.     

Control of neuromuscular stimulation for ambulation by complete paraplegics via artificial neural networks

The paper describes the application of a neural network (ANN) for controlling a functional neuromuscular stimulation (FNS) system to facilitate patient-responsive ambulation by paralyzed patients with traumatic, thoracic-level spinal cord injuries. The particular ANN that is employed is a modified Adaptive-Resonance-Theory (ART-1) network. It serves as a controller in an FNS system (the Parastep system) that is presently in use by approximately 500 patients worldwide (but still without ANN control) and which was the first and only FNS system approved by FDA. The proposed neural network discriminates above-lesion upper-trunk electromyographic (EMG) time series to activate standing and walking functions under FNS and controls FNS stimuli levels using response-EMG signals. For this particular application, several modifications are introduced into the standard ART-1 ANN. First, a modified on-line learning rule is proposed. The new rule assures bi-directional modification of the stored patterns and prevents noise interference. Second, a new reset rule is proposed, which prevents 'exact matching' when the input is a subset of the chosen pattern. A single ART-1-based structure is being applied to solving two problems, namely (1) signal pattern recognition and limb function determination, and (2) control of stimulation levels. This also facilitates ambulation of paraplegics under FNS, with adequate patient interaction in initial system training, retraining the network when needed, and in allowing patient's manual over-ride in the case of error, where any manual over-ride serves as a re-training input to the neural network. The ANN control facilitates continuous update of control settings during normal use, without formal retraining.     

Learning invariant face recognition from examples

Autonomous learning is demonstrated by living beings that learn visual invariances during their visual experience. Standard neural network models do not show this sort of learning. On the example of face recognition in different situations we propose a learning process that separates learning of the invariance proper from learning new instances of individuals. The invariance is learned by a set of examples called model, which contains instances of all situations. New instances are compared with these on the basis of rank lists, which allow generalization across situations. The result is also implemented as a spike-time-based neural network, which is shown to be robust against disturbances. The learning capability is demonstrated by recognition experiments on a set of standard face databases.     

Growing self-organizing trees for autonomous hierarchical clustering

This paper presents a new unsupervised learning method based on growing processes and autonomous self-assembly rules. This method, called Growing Self-organizing Trees (GSoT), can grow both network size and tree topology to represent the topological and hierarchical dataset organization, allowing a rapid and interactive visualization. Tree construction rules draw inspiration from elusive properties of biological organization to build hierarchical structures. Experiments conducted on real datasets demonstrate good GSoT performance and provide visual results that are generated during the training process.