Analysis of the process of visual pattern recognition by the neocognitron

A neural network model of visual pattern recognition called the “neocognitron,” was earlier proposed by the author. It is capable of deformation-invariant visual pattern recognition. After learning, it can recognize input patterns without being affected by deformation, changes in size, or shifts in position. This paper offers a mathematical analysis of the process of visual pattern recognition by the neocognitron. The neocognitron is a hierarchical multilayered network. Its initial stage is an input layer, and each succeeding stage has a layer of “S-cells” followed by a layer of “C-cells.” Thus, in the whole network, layers of S-cells and C-cells are arranged alternately. The process of feature extraction by an S-cell is analyzed mathematically in this paper, and the role of the C-cells in deformation-invariant pattern recognition is discussed.     

Learning of associative memory networks based upon cone-like domains of attraction

A learning algorithm for single layer perceptrons is proposed. First, a cone-like domain is derived such that all its elements can be recognized as a stored pattern in the perceptron network. The learning algorithm is obtained as a process that enlarges the cone-like domain. For autoassociative networks, it is shown that the cone-like domain becomes a domain of attraction for a stored pattern in the network. In this case, extended domains of attraction are also obtained by feeding the outputs of the network back to the input layer. In computer simulations, character recognition ability of the autoassociative network is examined.     

Shape Description and Recognition Method Inspired by the Primary Visual Cortex

The shape or contour of an object is usually stable and persistent, so it is a good basis for invariant recognition. Before this can be accomplished, two problems must be handled. The first is obtaining clean edges, and the other is organizing those edges into a structured form so they can be manipulated easily. Simple cells in the primary visual cortex are specialized for orientation detection. This neural mechanism can be stimulated using an array model. This model can produce a fairly clean set of lines, all in the form of vectors instead of pixels. These multiple orientation layers are then disintegrated from the original array, and a hierarchical partition tree can be created to re-organize them. Based on the similarity of the trees, a rough classification of objects can be realized. To enable maximum recognition, a moment-based measurement method was designed to describe the layout of active simple cells in each layer in detail. Then, a decision tree was produced using samples re-described using Hu’s moment invariants. This system takes into account more geometric information during the recognition process. The experimental results suggest that the representation efficiency enabled by simple cells, and their neural mechanism and the application of a multi-layered representation schema can simplify the algorithm. This further demonstrates the crucial role of simple cells in the visual processing path and shows that they can facilitate subsequent shape processing.     

A learning mechanism for invariant pattern recognition in neural networks

We show that a neural network with Hebbian learning and transmission delays will automatically perform invariant pattern recognition for a one-parameter transformation group. To do this it has to experience a learning phase in which static objects are presented as well as objects that are continuously undergoing small transformations. Our network is fully connected and starts with zero initial synapses so it does not require any a priori knowledge of the transformation group. From the information contained in the “moving” input, the network creates its internal representation of the transformation connecting the moving states. If the network cannot perform this transformation exactly, we show that in general the network representation will be a sensible approximation in terms of state overlaps. The limitation of our model is that we can implement only one-parameter transformation groups.     

Modeling cognitive and emotional processes: A novel neural network architecture

In our continuous attempts to model natural intelligence and emotions in machine learning, many research works emerge with different methods that are often driven by engineering concerns and have the common goal of modeling human perception in machines. This paper aims to go further in that direction by investigating the integration of emotion at the structural level of cognitive systems using the novel emotional DuoNeural Network (DuoNN). This network has hidden layer DuoNeurons, where each has two embedded neurons: a dorsal neuron and a ventral neuron for cognitive and emotional data processing, respectively. When input visual stimuli are presented to the DuoNN, the dorsal cognitive neurons process local features while the ventral emotional neurons process the entire pattern. We present the computational model and the learning algorithm of the DuoNN, the input information–cognitive and emotional–parallel streaming method, and a comparison between the DuoNN and a recently developed emotional neural network. Experimental results show that the DuoNN architecture, configuration, and the additional emotional information processing, yield higher recognition rates and faster learning and decision making.     

Training multi-layered neural network neocognitron

This paper proposes new learning rules suited for training multi-layered neural networks and applies them to the neocognitron. The neocognitron is a hierarchical multi-layered neural network capable of robust visual pattern recognition. It acquires the ability to recognize visual patterns through learning. For training intermediate layers of the hierarchical network of the neocognitron, we use a new learning rule named add-if-silent. By the use of the add-if-silent rule, the learning process becomes much simpler and more stable, and the computational cost for learning is largely reduced. Nevertheless, a high recognition rate can be kept without increasing the scale of the network. For the highest stage of the network, we use the method of interpolating-vector. We have previously reported that the recognition rate is greatly increased if this method is used during recognition. This paper proposes a new method of using it for both learning and recognition. Computer simulation demonstrates that the new neocognitron, which uses the add-if-silent and the interpolating-vector, produces a higher recognition rate for handwritten digits recognition with a smaller scale of the network than the neocognitron of previous versions.     

Use of non-uniform spatial blur for image comparison: symmetry axis extraction.

This paper shows that the introduction of non-uniform blur is very useful for comparing images, and proposes a neural network model that extracts axes of symmetry from visual patterns. The blurring operation greatly increases robustness against deformations and various kinds of noise, and largely reduces computational cost. Asymmetry between two groups of signals can be detected in a single action by the use of non-uniform blur having a cone-shaped distribution. The proposed model is a hierarchical multi-layered network, which consists of a contrast-extracting layer, edge-extracting layers (simple and complex types), and layers extracting symmetry axes. The model extracts oriented edges from an input image first, and then tries to extract axes of symmetry. The model checks conditions of symmetry, not directly from the oriented edges, but from a blurred version of the response of edge-extracting layer. The input patterns can be complicated line drawings, plane figures or gray-scaled natural images taken by CCD cameras.     

Three learning phases for radial-basis-function networks.

In this paper, learning algorithms for radial basis function (RBF) networks are discussed. Whereas multilayer perceptrons (MLP) are typically trained with backpropagation algorithms, starting the training procedure with a random initialization of the MLP's parameters, an RBF network may be trained in many different ways. We categorize these RBF training methods into one-, two-, and three-phase learning schemes. Two-phase RBF learning is a very common learning scheme. The two layers of an RBF network are learnt separately; first the RBF layer is trained, including the adaptation of centers and scaling parameters, and then the weights of the output layer are adapted. RBF centers may be trained by clustering, vector quantization and classification tree algorithms, and the output layer by supervised learning (through gradient descent or pseudo inverse solution). Results from numerical experiments of RBF classifiers trained by two-phase learning are presented in three completely different pattern recognition applications: (a) the classification of 3D visual objects; (b) the recognition hand-written digits (2D objects); and (c) the categorization of high-resolution electrocardiograms given as a time series (ID objects) and as a set of features extracted from these time series. In these applications, it can be observed that the performance of RBF classifiers trained with two-phase learning can be improved through a third backpropagation-like training phase of the RBF network, adapting the whole set of parameters (RBF centers, scaling parameters, and output layer weights) simultaneously. This, we call three-phase learning in RBF networks. A practical advantage of two- and three-phase learning in RBF networks is the possibility to use unlabeled training data for the first training phase. Support vector (SV) learning in RBF networks is a different learning approach. SV learning can be considered, in this context of learning, as a special type of one-phase learning, where only the output layer weights of the RBF network are calculated, and the RBF centers are restricted to be a subset of the training data. Numerical experiments with several classifier schemes including k-nearest-neighbor, learning vector quantization and RBF classifiers trained through two-phase, three-phase and support vector learning are given. The performance of the RBF classifiers trained through SV learning and three-phase learning are superior to the results of two-phase learning, but SV learning often leads to complex network structures, since the number of support vectors is not a small fraction of the total number of data points.     

Stability of generalized topographic mappings between cell layers through correlational learning.

We propose a simple topographic mapping formation model from a cell layer to a cell layer. Our model is a discrete one in that the state value of input and output cells takes 0 or 1 and input and output layers are represented by undirected graphs. A binary input pattern can be given to the network consisting of input and output cell layers. Such an input pattern can be represented by a subset of input cells. That is, a state value of an input cell takes 1 if a cell belongs to the subset, otherwise, a state value of an input cell is 0. Such a definition of an input pattern does not necessarily assume a short-range excitatory mechanism in an input layer. Thus, a topographic mapping described in this model is a map, which preserves the input pattern relation. By using the concept of input pattern separability, we showed an existence condition of certain learning rules, which are correlational. We have paid special attention to such correlational type learning rules, and have shown under the rules that topographic mappings are the only stable ones. As to the non-correlational learning rules, we also investigate the stability of generated mappings.     

Multilayer in-place learning networks for modeling functional layers in the laminar cortex

Currently, there is a lack of general-purpose in-place learning networks that model feature layers in the cortex. By “general-purpose” we mean a general yet adaptive high-dimensional function approximator. In-place learning is a biological concept rooted in the genomic equivalence principle, meaning that each neuron is fully responsible for its own learning in its environment and there is no need for an external learner. Presented in this paper is the Multilayer In-place Learning Network (MILN) for this ambitious goal. Computationally, in-place learning provides unusually efficient learning algorithms whose simplicity, low computational complexity, and generality are set apart from typical conventional learning algorithms. Based on the neuroscience literature, we model the layer 4 and layer 2/3 as the feature layers in the 6-layer laminar cortex, with layer 4 using unsupervised learning and layer 2/3 using supervised learning. As a necessary requirement for autonomous mental development, MILN generates invariant neurons in different layers, with increasing invariance from earlier to later layers and the total invariance in the last motor layer. Such self-generated invariant representation is enabled mainly by descending (top-down) connections. The self-generated invariant representation is used as intermediate representations for learning later tasks in open-ended development.     

How Deep Should be the Depth of Convolutional Neural Networks: a Backyard Dog Case Study

The work concerns the problem of reducing a pre-trained deep neuronal network to a smaller network, with just few layers, whilst retaining the network’s functionality on a given task. In this particular case study, we are focusing on the networks developed for the purposes of face recognition. The proposed approach is motivated by the observation that the aim to deliver the highest accuracy possible in the broadest range of operational conditions, which many deep neural networks models strive to achieve, may not necessarily be always needed, desired or even achievable due to the lack of data or technical constraints. In relation to the face recognition problem, we formulated an example of such a use case, the ‘backyard dog’ problem. The ‘backyard dog’, implemented by a lean network, should correctly identify members from a limited group of individuals, a ‘family’, and should distinguish between them. At the same time, the network must produce an alarm to an image of an individual who is not in a member of the family, i.e. a ‘stranger’. To produce such a lean network, we propose a network shallowing algorithm. The algorithm takes an existing deep learning model on its input and outputs a shallowed version of the model. The algorithm is non-iterative and is based on the advanced supervised principal component analysis. Performance of the algorithm is assessed in exhaustive numerical experiments. Our experiments revealed that in the above use case, the ‘backyard dog’ problem, the method is capable of drastically reducing the depth of deep learning neural networks, albeit at the cost of mild performance deterioration. In this work, we proposed a simple non-iterative method for shallowing down pre-trained deep convolutional networks. The method is generic in the sense that it applies to a broad class of feed-forward networks, and is based on the advanced supervise principal component analysis. The method enables generation of families of smaller-size shallower specialized networks tuned for specific operational conditions and tasks from a single larger and more universal legacy network.

     

Dolphin Swarm Extreme Learning Machine

As a novel learning algorithm for a single hidden-layer feedforward neural network, the extreme learning machine has attracted much research attention for its fast training speed and good generalization performances. Instead of iteratively tuning the parameters, the extreme machine can be seen as a linear optimization problem by randomly generating the input weights and hidden biases. However, the random determination of the input weights and hidden biases may bring non-optimal parameters, which have a negative impact on the final results or need more hidden nodes for the neural network. To overcome the above drawbacks caused by the non-optimal input weights and hidden biases, we propose a new hybrid learning algorithm named dolphin swarm algorithm extreme learning machine adopting the dolphin swarm algorithm to optimize the input weights and hidden biases efficiently. Each set of input weights and hidden biases is encoded into one vector, namely the dolphin. The dolphins are evaluated by root mean squared error and updated by the four pivotal phases of the dolphin swarm algorithm. Eventually, we will obtain an optimal set of input weights and hidden biases. To evaluate the effectiveness of our method, we compare the proposed algorithm with the standard extreme learning machine and three state-of-the-art methods, which are the particle swarm optimization extreme learning machine, evolutionary extreme learning machine, and self-adaptive evolutionary extreme learning machine, under 13 benchmark datasets obtained from the University of California Irvine Machine Learning Repository. The experimental results demonstrate that the proposed method can achieve superior generalization performances than all the compared algorithms.     

A neural model of quantity discrimination

A neural network model is proposed with the ability to extract abstract numerical representation from visual input. It simulates properties of a number detection system which is hypothesized to underlie simple language-independent numerical abilities. The network has three layers where the first layer computes the sum of the nearest neighbour inputs. The first layer is also augmented with multiplicative gating and gradient tonic activation which prevents interference. The second layer implements local lateral inhibition which enables a single node to represent a single object. The third layer exhibits number-tuning similar to recently described responses of neurons in the prefrontal cortex. Computer simulations showed that network response does not depend on visual attributes like the object's size, position or shape. The model is based on several biophysical mechanisms such as multiplicative interaction on dendrites, independent processing on different dendritic branches and disinhibition by glutamate spill-over on kainate receptors on inhibitory axons.     

Optimal unsupervised learning in a single-layer linear feedforward neural network

A new approach to unsupervised learning in a single-layer linear feedforward neural network is discussed. An optimality principle is proposed which is based upon preserving maximal information in the output units. An algorithm for unsupervised learning based upon a Hebbian learning rule, which achieves the desired optimality is presented. The algorithm finds the eigenvectors of the input correlation matrix, and it is proven to converge with probability one. An implementation which can train neural networks using only local “synaptic” modification rules is described. It is shown that the algorithm is closely related to algorithms in statistics (Factor Analysis and Principal Components Analysis) and neural networks (Self-supervised Backpropagation, or the “encoder” problem). It thus provides an explanation of certain neural network behavior in terms of classical statistical techniques. Examples of the use of a linear network for solving image coding and texture segmentation problems are presented. Also, it is shown that the algorithm can be used to find “visual receptive fields” which are qualitatively similar to those found in primate retina and visual cortex.     

Self-organization of shift-invariant receptive fields

This paper proposes a new learning rule by which cells with shift-invariant receptive fields are self-organized. With this learning rule, cells similar to simple and complex cells in the primary visual cortex are generated in a network. To demonstrate the new learning rule, we simulate a three-layered network that consists of an input layer (or the retina), a layer of S-cells (or simple cells), and a layer of C-cells (or complex cells). During the learning, straight lines of various orientations sweep across the input layer. Here both S- and C-cells are created through competition. Although S-cells compete depending on their instantaneous outputs, C-cells compete depending on the traces (or temporal averages) of their outputs. For the self-organization of S-cells, only winner S-cells increase their input connections in a similar way to that for the neocognitron. In other words, the winner S-cells have LTP (long term potentiation) in their input connections. For the self-organization of C-cells, however, loser C-cells decrease their input connections (LTD=long term depression), while winners increase their input connections (LTP). Here both S- and C-cells are accompanied by inhibitory cells. Modification of inhibitory connections together with excitatory connections is important for creation of C-cells as well as S-cells.     

Density codes and shape spaces.

This paper presents an algorithm that allows for encoding probability density functions associated to samples of points of R(n). The resulting code is a sequence of points of R(n) whose density function approximates that of the set of data points. However, contrarily to sampled data points, code points associated to two different density functions can be matched, which allows to efficiently compare such functions. Moreover, the comparison of two codes can be made invariant to a wide variety of geometrical transformations of the support coordinates, provided that the Jacobian matrix of the transformation be everywhere triangular, with a strictly positive diagonal. Such invariances are commonly encountered in visual shape recognition, for example. Thus, using this tool, one can build spaces of shapes that are suitable input spaces for pattern recognition and pattern analysis neural networks. Moreover, a parallel neural implementation of the encoding algorithm is available for 2D image data.     

Neocognitron and the Map Transformation Cascade

Based on our observations of the working principles of the archetypal hierarchical neural network, Neocognitron, we propose a simplified model which we call the Map Transformation Cascade. The least complex Map Transformation Cascade can be understood as a sequence of filters, which maps and transforms the input pattern into a space where patterns in the same class are close. The output of the filters is then passed to a simple classifier, which yields a classification for the input pattern. Instead of a specifically crafted learning algorithm, the Map Transformation Cascade separates two different learning needs: Information reduction, where a clustering algorithm is more suitable (e.g., K-Means) and classification, where a supervised classifier is more suitable (e.g., nearest neighbor method). The performance of the proposed model is analyzed in handwriting recognition. The Map Transformation Cascade achieved performance similar to that of Neocognitron.     

The Role of Horizontal Connections in Generating Long Receptive Fields in the Cat Visual Cortex

The cells in the primary visual cortex possess numerous functional properties that are more complex and varied than those seen in the cortical input. These properties result from the network of intrinsic cortical connections running across the cortical layers and between cortical columns. In the current study we relate the long receptive fields that are characteristic of layer 6 cells to the input that these cells receive from layer 5. The axons of layer 5 pyramidal cells project over long distances within layer 6, enabling layer 6 cells to collect input from regions of cortex representing large parts of the visual field. When layer 5 was locally inactivated by injection of the inhibitory transmitter GABA, layer 6 cells lost sensitivity over the portion of their receptive fields corresponding to the inactivated region of layer 5. This suggests that the extensive convergence in the projection from layer 5 to layer 6 is responsible for generating the long receptive fields characteristic of the layer 6 cells.     

Self-organizing neural projections.

The Self-Organizing Map (SOM) algorithm was developed for the creation of abstract-feature maps. It has been accepted widely as a data-mining tool, and the principle underlying it may also explain how the feature maps of the brain are formed. However, it is not correct to use this algorithm for a model of pointwise neural projections such as the somatotopic maps or the maps of the visual field, first of all, because the SOM does not transfer signal patterns: the winner-take-all function at its output only defines a singular response. Neither can the original SOM produce superimposed responses to superimposed stimulus patterns. This presentation introduces a new self-organizing system model related to the SOM that has a linear transfer function for patterns and combinations of patterns all the time. Starting from a randomly interconnected pair of neural layers, and using random mixtures of patterns for training, it creates a pointwise-ordered projection from the input layer to the output layer. If the input layer consists of feature detectors, the output layer forms a feature map of the inputs.