Training multi-layered neural network neocognitron

This paper proposes new learning rules suited for training multi-layered neural networks and applies them to the neocognitron. The neocognitron is a hierarchical multi-layered neural network capable of robust visual pattern recognition. It acquires the ability to recognize visual patterns through learning. For training intermediate layers of the hierarchical network of the neocognitron, we use a new learning rule named add-if-silent. By the use of the add-if-silent rule, the learning process becomes much simpler and more stable, and the computational cost for learning is largely reduced. Nevertheless, a high recognition rate can be kept without increasing the scale of the network. For the highest stage of the network, we use the method of interpolating-vector. We have previously reported that the recognition rate is greatly increased if this method is used during recognition. This paper proposes a new method of using it for both learning and recognition. Computer simulation demonstrates that the new neocognitron, which uses the add-if-silent and the interpolating-vector, produces a higher recognition rate for handwritten digits recognition with a smaller scale of the network than the neocognitron of previous versions.     

Short utterance recognition using a network with minimum training

A feedforward network is used to recognize short, digitized, isolated utterances. A high, multispeaker recognition rate is achieved with a small vocabulary with a single training utterance. This approach makes use of the pattern recognition property of the network architecture to classify different temporal patterns in the multidimensional feature space. The network recognizes the utterances without the need of segmentation, phoneme identification, or time alignment. We train the network with four words spoken by one single speaker. The network is then able to recognize 20 tokens spoken by 5 other speakers. We repeat the above training and testing procedure using a different speaker's utterances for training each time. The overall accuracy is 97.5%. We compare this approach to the traditional dynamic programming (DP) approach, and find that DP with slope constraints of 0 and 1 achieve 98.5% and 85% accuracies respectively. Finally we validate out statistics by training and testing the network of a four-word subset of the Texas Instruments (Tl) isolated word database. The accuracy with this vocabulary exceeds 96%. By doubling the size of the training set, the accuracy is raised to 98%. Using a suitable threshold, we are able to raise the accuracy of one network from 87% to 98.5%. Thresholding applied to all networks would then raise the overall accuracy to well over 99%.

This technique is especially promising because of the low overhead and computational requirements, which make it suitable for a low cost, portable, command recognition type of application.     

Classification of 2-dimensional array patterns: Assembling many small neural networks is better than using a large one

In many pattern classification/recognition applications of artificial neural networks, an object to be classified is represented by a fixed sized 2-dimensional array of uniform type, which corresponds to the cells of a 2-dimensional grid of the same size. A general neural network structure, called an undistricted neural network, which takes all the elements in the array as inputs could be used for problems such as these. However, a districted neural network can be used to reduce the training complexity. A districted neural network usually consists of two levels of sub-neural networks. Each of the lower level neural networks, called a regional sub-neural network, takes the elements in a region of the array as its inputs and is expected to output a temporary class label, called an individual opinion, based on the partial information of the entire array. The higher level neural network, called an assembling sub-neural network, uses the outputs (opinions) of regional sub-neural networks as inputs, and by consensus derives the label decision for the object. Each of the sub-neural networks can be trained separately and thus the training is less expensive. The regional sub-neural networks can be trained and performed in parallel and independently, therefore a high speed can be achieved. We prove theoretically in this paper, using a simple model, that a districted neural network is actually more stable than an undistricted neural network in noisy environments. We conjecture that the result is valid for all neural networks.This theory is verified by experiments involving gender classification and human face recognition. We conclude that a districted neural network is highly recommended for neural network applications in recognition or classification of 2-dimensional array patterns in highly noisy environments.     

Three learning phases for radial-basis-function networks.

In this paper, learning algorithms for radial basis function (RBF) networks are discussed. Whereas multilayer perceptrons (MLP) are typically trained with backpropagation algorithms, starting the training procedure with a random initialization of the MLP's parameters, an RBF network may be trained in many different ways. We categorize these RBF training methods into one-, two-, and three-phase learning schemes. Two-phase RBF learning is a very common learning scheme. The two layers of an RBF network are learnt separately; first the RBF layer is trained, including the adaptation of centers and scaling parameters, and then the weights of the output layer are adapted. RBF centers may be trained by clustering, vector quantization and classification tree algorithms, and the output layer by supervised learning (through gradient descent or pseudo inverse solution). Results from numerical experiments of RBF classifiers trained by two-phase learning are presented in three completely different pattern recognition applications: (a) the classification of 3D visual objects; (b) the recognition hand-written digits (2D objects); and (c) the categorization of high-resolution electrocardiograms given as a time series (ID objects) and as a set of features extracted from these time series. In these applications, it can be observed that the performance of RBF classifiers trained with two-phase learning can be improved through a third backpropagation-like training phase of the RBF network, adapting the whole set of parameters (RBF centers, scaling parameters, and output layer weights) simultaneously. This, we call three-phase learning in RBF networks. A practical advantage of two- and three-phase learning in RBF networks is the possibility to use unlabeled training data for the first training phase. Support vector (SV) learning in RBF networks is a different learning approach. SV learning can be considered, in this context of learning, as a special type of one-phase learning, where only the output layer weights of the RBF network are calculated, and the RBF centers are restricted to be a subset of the training data. Numerical experiments with several classifier schemes including k-nearest-neighbor, learning vector quantization and RBF classifiers trained through two-phase, three-phase and support vector learning are given. The performance of the RBF classifiers trained through SV learning and three-phase learning are superior to the results of two-phase learning, but SV learning often leads to complex network structures, since the number of support vectors is not a small fraction of the total number of data points.     

Fat neural network for recognition of position-normalised objects

The design of a recognition system for natural objects is difficult, mainly because such objects are subject to a strong variability that cannot be easily modelled: planktonic species possess such highly variable forms. Existing plankton recognition systems usually comprise feature extraction processing upstream of a classifier. Drawbacks of such an approach are that the design of relevant feature extraction processes may be very difficult, especially if classes are numerous and if intra-class variability is high, so that the system becomes specific to the problem for which features have been tuned. The opposite course that we take is based on a structured multi-layer neural network with no shared weights, which generates its own features during training. Such a large parameterised—fat—network exhibits good generalisation capabilities for pattern recognition problems dealing with position-normalised objects, even with as many as one thousand weights as training examples. The advantage of such large networks, in terms of generalisation efficiency, adaptability and classification time, is demonstrated by applying the network to three plankton recognition and face recognition problems. Its ability to perform good generalisation with few training examples, but many weights, is an open theoretical problem.     

To spike or not to spike: A probabilistic spiking neuron model

Spiking neural networks (SNN) are promising artificial neural network (ANN) models as they utilise information representation as trains of spikes, that adds new dimensions of time, frequency and phase to the structure and the functionality of ANN. The current SNN models though are deterministic, that restricts their applications for large scale engineering and cognitive modelling of stochastic processes. This paper proposes a novel probabilistic spiking neuron model (pSNM) and suggests ways of building pSNN for a wide range of applications including classification, string pattern recognition and associative memory. It also extends previously published computational neurogenetic models.     

Pattern classification by a condensed neural network.

Neural networks have come to the fore as potent pattern classifiers. More amenable to parallel computation, they are much faster than the nearest neighbor classifier (NN), which, however, has distinctly outperformed them in several applications. The purpose of this study is to investigate a condensed neural network that combines the classification speed of neural networks and the low error rate of the nearest neighbor classifier. This condensed network is a fast, accurate classifier of simple architecture and function: it consists of a set of generalized perceptrons that draw maximal hyperspherical boundaries centered on patterns of memory units, each circumscribing reference patterns of a single category. The generalized perceptrons carry out classification, assisted by sporadic nearest neighbor matching to patterns of a small reference set. We compare the condensed network to a high performance neural network pattern classifier (Kohonen) and to NN in experiments on hand-printed character recognition.     

Learning by doing? The effect of gestures on implicit retrieval of newly acquired words

Meaningful gestures enhance speech comprehensibility. However, their role during novel-word acquisition remains elusive. Here we investigate how meaningful versus meaningless gestures impact on novel-word learning and contrast these conditions to a purely verbal training. After training, neuronal processing of the novel words was assessed by blood-oxygen-level-dependent functional magnetic resonance imaging (BOLD-fMRI), disclosing that networks affording retrieval differ depending on the training condition. Over 3 days participants learned pseudowords for common objects (e.g., /klira/ -cap). For training they repeated the novel word while performing (i) an iconic, (ii) a grooming or (iii) no gesture. For the two conditions involving gestures, these were either actively repeated or passively observed during training. Behaviorally no substantial differences between the five different training conditions were found while fMRI disclosed differential networks affording implicit retrieval of the learned pseudowords depending on the training procedure. Most notably training with actively performed iconic gestures yielded larger activation in a semantic network comprising left inferior frontal (BA47) and inferior temporal gyri. Additionally hippocampal activation was stronger for all trained compared to unknown pseudowords of identical structure. The behavioral results challenge the generality of an ‘enactment-effect’ for single word learning. Imaging results, however, suggest that actively performed meaningful gestures lead to a deeper semantic encoding of novel words. The findings are discussed regarding their implications for theoretical accounts and for empirical approaches of gesture-based strategies in language (re)learning.     

Hierarchical classification with a competitive evolutionary neural tree

A new, dynamic, tree structured network, the Competitive Evolutionary Neural Tree (CENT) is introduced. The network is able to provide a hierarchical classification of unlabelled data sets. The main advantage that the CENT offers over other hierarchical competitive networks is its ability to self determine the number, and structure, of the competitive nodes in the network, without the need for externally set parameters. The network produces stable classificatory structures by halting its growth using locally calculated heuristics. The results of network simulations are presented over a range of data sets, including Anderson’s IRIS data set. The CENT network demonstrates its ability to produce a representative hierarchical structure to classify a broad range of data sets.     

ARTMAP neural networks for information fusion and data mining: map production and target recognition methodologies.

The Sensor Exploitation Group of MIT Lincoln Laboratory incorporated an early version of the ARTMAP neural network as the recognition engine of a hierarchical system for fusion and data mining of registered geospatial images. The Lincoln Lab system has been successfully fielded, but is limited to target/non-target identifications and does not produce whole maps. Procedures defined here extend these capabilities by means of a mapping method that learns to identify and distribute arbitrarily many target classes. This new spatial data mining system is designed particularly to cope with the highly skewed class distributions of typical mapping problems. Specification of canonical algorithms and a benchmark testbed has enabled the evaluation of candidate recognition networks as well as pre- and post-processing and feature selection options. The resulting mapping methodology sets a standard for a variety of spatial data mining tasks. In particular, training pixels are drawn from a region that is spatially distinct from the mapped region, which could feature an output class mix that is substantially different from that of the training set. The system recognition component, default ARTMAP, with its fully specified set of canonical parameter values, has become the a priori system of choice among this family of neural networks for a wide variety of applications.     

Machine learning identifies unaffected first‐degree relatives with functional network patterns and cognitive impairment similar to those of schizophrenia patients

Schizophrenia (SCZ) patients and their unaffected first‐degree relatives (FDRs) share similar functional neuroanatomy. However, it remains largely unknown to what extent unaffected FDRs with functional neuroanatomy patterns similar to patients can be identified at an individual level. In this study, we used a multivariate pattern classification method to learn informative large‐scale functional networks (FNs) and build classifiers to distinguish 32 patients from 30 healthy controls and to classify 34 FDRs as with or without FNs similar to patients. Four informative FNs—the cerebellum, default mode network (DMN), ventral frontotemporal network, and posterior DMN with parahippocampal gyrus—were identified based on a training cohort and pattern classifiers built upon these FNs achieved a correct classification rate of 83.9% (sensitivity 87.5%, specificity 80.0%, and area under the receiver operating characteristic curve [AUC] 0.914) estimated based on leave‐one‐out cross‐validation for the training cohort and a correct classification rate of 77.5% (sensitivity 72.5%, specificity 82.5%, and AUC 0.811) for an independent validation cohort. The classification scores of the FDRs and patients were negatively correlated with their measures of cognitive function. FDRs identified by the classifiers as having SCZ patterns were similar to the patients, but significantly different from the controls and FDRs with normal patterns in terms of their cognitive measures. These results demonstrate that the pattern classifiers built upon the informative FNs can serve as biomarkers for quantifying brain alterations in SCZ and help to identify FDRs with FN patterns and cognitive impairment similar to those of SCZ patients.     

Organization of the state space of a simple recurrent network before and after training on recursive linguistic structures.

Recurrent neural networks are often employed in the cognitive science community to process symbol sequences that represent various natural language structures. The aim is to study possible neural mechanisms of language processing and aid in development of artificial language processing systems. We used data sets containing recursive linguistic structures and trained the Elman simple recurrent network (SRN) for the next-symbol prediction task. Concentrating on neuron activation clusters in the recurrent layer of SRN we investigate the network state space organization before and after training. Given a SRN and a training stream, we construct predictive models, called neural prediction machines, that directly employ the state space dynamics of the network. We demonstrate two important properties of representations of recursive symbol series in the SRN. First, the clusters of recurrent activations emerging before training are meaningful and correspond to Markov prediction contexts. We show that prediction states that naturally arise in the SRN initialized with small random weights approximately correspond to states of Variable Memory Length Markov Models (VLMM) based on individual symbols (i.e. words). Second, we demonstrate that during training, the SRN reorganizes its state space according to word categories and their grammatical subcategories, and the next-symbol prediction is again based on the VLMM strategy. However, after training, the prediction is based on word categories and their grammatical subcategories rather than individual words. Our conclusion holds for small depths of recursions that are comparable to human performances. The methods of SRN training and analysis of its state space introduced in this paper are of a general nature and can be used for investigation of processing of any other symbol time series by means of SRN.     

Self partitioning neural networks for target recognition

Automatic target recognition (ATR) is a domain in which the neural network technology has been applied with limited success. The domain is characterized by large training sets with dissimilar target images carrying conflicting information. This paper presents a novel method for quantifying the degree of non-cooperation that exists among the target members of the training set. Both the network architecture and the training algorithm are considered in the computation of the non-cooperation measures. Based on these measures, the self partitioning neural network (SPNN) approach partitions the target vectors into an appropriate number of groups and trains one subnetwork to recognize the targets in each group. A fusion network combines the outputs of the subnetworks to produce the final response. This method automatically determines the number of subnetworks needed without excessive computation. The subnetworks are simple with only one hidden layer and one unit in the output layer. They are topologically identical to one another. The simulation results indicate that the method is robust and capable of self organization to overcome the ill effects of the non-cooperating targets in the training set. The self partitioning approach improves the classification accuracy and reduces the training time of neural networks significantly. It is also shown that a trained self partitioning neural network is capable of learning new training vectors without retraining on the combined training set (i.e., the training set consisting of the previous and newly acquired training vectors).     

A maze learning comparison of Elman,long short-term memory,and Mona neural networks

This study compares the maze learning performance of three artificial neural network architectures: an Elman recurrent neural network, a long short-term memory (LSTM) network, and Mona, a goal-seeking neural network. The mazes are networks of distinctly marked rooms randomly interconnected by doors that open probabilistically. The mazes are used to examine two important problems related to artificial neural networks: (1) the retention of long-term state information and (2) the modular use of learned information. For the former, mazes impose a context learning demand: at the beginning of the maze, an initial door choice forms a context that must be remembered until the end of the maze, where the same numbered door must be chosen again in order to reach the goal. For the latter, the effect of modular and non-modular training is examined. In modular training, the door associations are trained in separate trials from the intervening maze paths, and only presented together in testing trials. All networks performed well on mazes without the context learning requirement. The Mona and LSTM networks performed well on context learning with non-modular training; the Elman performance degraded as the task length increased. Mona also performed well for modular training; both the LSTM and Elman networks performed poorly with modular training.     

Small-world networks in neuronal populations: A computational perspective

The analysis of the brain in terms of integrated neural networks may offer insights on the reciprocal relation between structure and information processing. Even with inherent technical limits, many studies acknowledge neuron spatial arrangements and communication modes as key factors.In this perspective, we investigated the functional organization of neuronal networks by explicitly assuming a specific functional topology, the small-world network. We developed two different computational approaches. Firstly, we asked whether neuronal populations actually express small-world properties during a definite task, such as a learning task. For this purpose we developed the Inductive Conceptual Network (ICN), which is a hierarchical bio-inspired spiking network, capable of learning invariant patterns by using variable-order Markov models implemented in its nodes. As a result, we actually observed small-world topologies during learning in the ICN. Speculating that the expression of small-world networks is not solely related to learning tasks, we then built a de facto network assuming that the information processing in the brain may occur through functional small-world topologies. In this de facto network, synchronous spikes reflected functional small-world network dependencies. In order to verify the consistency of the assumption, we tested the null-hypothesis by replacing the small-world networks with random networks. As a result, only small world networks exhibited functional biomimetic characteristics such as timing and rate codes, conventional coding strategies and neuronal avalanches, which are cascades of bursting activities with a power-law distribution.Our results suggest that small-world functional configurations are liable to underpin brain information processing at neuronal level.     

Semantic priming in thought disordered schizophrenic patients

Groups of thought disordered (TD) and non-thought disordered (NTD) schizophrenic patients, unipolar affective patients and normal controls performed a lexical decision task involving the recognition of words immediately preceded (primed) by either an associated or an unrelated word. Significant increments in recognition speed in the associated prime condition were found in all groups, with significantly greater gain by TD schizophrenics than by others. These findings are consistent with network models of associational activation and lend support to an attentional deficit hypothesis for schizophrenic language functioning.     

Pattern recognition of the electroencephalogram by artificial neural networks

A back-propagation network was trained to recognize high voltage spike-wave spindle (HVS) patterns in the rat, a rodent model of human petit mal epilepsy. The spontaneously occurring HVSs were examined in 137 rats of the Fisher 344 and Brown Norway strains and their F1, F2 and backcross hybrids. Neocortical EEG and movement of the rat were recorded for 12 night hours in each animal and analog data were filtered (low cut: 1 Hz; high cut: 50 Hz) and sampled at 100 Hz with 12 bit precision. A training data set was generated by manually marking durations of HVS epochs in 16 representative animals selected from each group. Training data were presented to back-propagation networks with variable numbers of input, hidden and output cells. The performance of different types of networks was first examined with the training samples and then the best configuration was tested on novel sets of the EEG data. FFT transformation of EEG significantly improved the pattern recognition ability of the network. With the most effective configuration (16 input; 19 hidden; 1 output cells) the summed squared error dropped by 80% as compared with that of the initial random weights. When testing the network with new patterns the manual and automatic evaluations were compared quantitatively. HVSs which were detected properly by the network reached 93–99% of the manually marked HVS patterns, while falsely detected events (non-HVS, artifacts) varied between 18% and 40%. These findings demonstrate the utility of back-propagation networks in automatic recognition of EEG patterns.     

Categorization and decision-making in a neurobiologically plausible spiking network using a STDP-like learning rule

Understanding how the human brain is able to efficiently perceive and understand a visual scene is still a field of ongoing research. Although many studies have focused on the design and optimization of neural networks to solve visual recognition tasks, most of them either lack neurobiologically plausible learning rules or decision-making processes. Here we present a large-scale model of a hierarchical spiking neural network (SNN) that integrates a low-level memory encoding mechanism with a higher-level decision process to perform a visual classification task in real-time. The model consists of Izhikevich neurons and conductance-based synapses for realistic approximation of neuronal dynamics, a spike-timing-dependent plasticity (STDP) synaptic learning rule with additional synaptic dynamics for memory encoding, and an accumulator model for memory retrieval and categorization. The full network, which comprised 71,026 neurons and approximately 133 million synapses, ran in real-time on a single off-the-shelf graphics processing unit (GPU). The network was constructed on a publicly available SNN simulator that supports general-purpose neuromorphic computer chips. The network achieved 92% correct classifications on MNIST in 100 rounds of random sub-sampling, which is comparable to other SNN approaches and provides a conservative and reliable performance metric. Additionally, the model correctly predicted reaction times from psychophysical experiments. Because of the scalability of the approach and its neurobiological fidelity, the current model can be extended to an efficient neuromorphic implementation that supports more generalized object recognition and decision-making architectures found in the brain.     

Predicting responses of nonlinear neurons in monkey striate cortex to complex patterns.

The overwhelming majority of neurons in primate visual cortex are nonlinear. For those cells, the techniques of linear system analysis, used with some success to model retinal ganglion cells and striate simple cells, are of limited applicability. As a start toward understanding the properties of nonlinear visual neurons, we have recorded responses of striate complex cells to hundreds of images, including both simple stimuli (bars and sinusoids) as well as complex stimuli (random textures and 3-D shaded surfaces). The latter set tended to give the strongest response. We created a neural network model for each neuron using an iterative optimization algorithm. The recorded responses to some stimulus patterns (the training set) were used to create the model, while responses to other patterns were reserved for testing the networks. The networks predicted recorded responses to training set patterns with a median correlation of 0.95. They were able to predict responses to test stimuli not in the training set with a correlation of 0.78 overall, and a correlation of 0.65 for complex stimuli considered alone. Thus, they were able to capture much of the input/output transfer function of the neurons, even for complex patterns. Examining connection strengths within each network, different parts of the network appeared to handle information at different spatial scales. To gain further insights, the network models were inverted to construct "optimal" stimuli for each cell, and their receptive fields were mapped with high-resolution spots. The receptive field properties of complex cells could not be reduced to any simpler mathematical formulation than the network models themselves.