Bi-directional computing architecture for time series prediction.

A number of neural network models and training procedures for time series prediction have been proposed in the technical literature. These models studied for different time-variant data sets have typically used uni-directional computation flow or its modifications. In this study, on the contrary, the concept of bi-directional computational style is proposed and applied to prediction tasks. A bi-directional neural network model consists of two subnetworks performing two types of signal transformations bi-directionally. The networks also receive complementary signals from each other through mutual connections. The model not only deals with the conventional future prediction task, but also with the past prediction, an additional task from the viewpoint of the conventional approach. An improvement of the performance is achieved through making use of the future-past information integration. Since the coupling effects help the proposed model improve its performance, it is found that the prediction score is better than with the traditional uni-directional method. The bi-directional predicting architecture has been found to perform better than the conventional one when tested with standard benchmark sunspots data.     

Unsupervised classification of neural spikes with a hybrid multilayer artificial neural network

The understanding of the brain structure and function and its computational style is one of the biggest challenges both in Neuroscience and Neural Computation. In order to reach this and to test the predictions of neural network modeling, it is necessary to observe the activity of neural populations. In this paper we propose a hybrid modular computational system for the spike classification of multiunits recordings. It works with no knowledge about the waveform, and it consists of two moduli: a Preprocessing (Segmentation) module, which performs the detection and centering of spike vectors using programmed computation; and a Processing (Classification) module, which implements the general approach of neural classification: feature extraction, clustering and discrimination, by means of a hybrid unsupervised multilayer artificial neural network (HUMANN). The operations of this artificial neural network on the spike vectors are: (i) compression with a Sanger Layer from 70 points vector to five principal component vector; (ii) their waveform is analyzed by a Kohonen layer; (iii) the electrical noise and overlapping spikes are rejected by a previously unreported artificial neural network named Tolerance layer; and (iv) finally the spikes are labeled into spike classes by a Labeling layer. Each layer of the system has a specific unsupervised learning rule that progressively modifies itself until the performance of the layer has been automatically optimized. The procedure showed a high sensitivity and specificity also when working with signals containing four spike types.     

A method to assess the functional impact of cerebral connections on target populations of neurons

We describe an innovative and tested approach combining two individually potent techniques to visualize simultaneously the functional impact of multiple projections on target populations of neurons in the brain. The rationale is simple: silence a defined set of efferent projections from one cortical region using cooling deactivation and then measure the impact of the deactivation on activities in multiple target regions using 2-deoxyglucose (2DG). This is a straightforward and sound approach because 2DG uptake by neurons reflects levels of underlying neural activity. All distant modifications evoked by the silencing of the set of efferent projections are examined in anatomical tissue and simultaneously for the multiple target sites to provide a global view of the functional impacts of the set of projections on the targets. With this method, downward adjustments of 2DG uptake levels identify removals of net excitatory signals, whereas upward adjustments identify net removals of suppressive influences. Future possible uses and modifications of the technique, including optical imaging, are discussed. Overall, the technique has the potential to provide fundamental, new measures on cerebral network interactions that both complement and extend current static models of cerebral networks and electrophysiological measures of functional impacts on individual neurons.     

Using Spatio-temporal Correlations to Learn Invariant Object Recognition

A competitive network is described which learns to classify objects on the basis of temporal as well as spatial correlations. This is achieved by using a Hebb-like learning rule which is dependent upon prior as well as current neural activity. The rule is shown to be capable of outperforming a supervised rule on the cross validation test of an invariant character recognition task, given a relatively small training set. It is also shown to outperform the supervised version of Fukushima's Neocognitron (Fukushima, 1980), on a larger training set. Copyright © 1996 Elsevier Science Ltd.     

Quick fuzzy backpropagation algorithm.

A modification of the fuzzy backpropagation (FBP) algorithm called QuickFBP algorithm is proposed, where the computation of the net function is significantly quicker. It is proved that the FBP algorithm is of exponential time complexity, while the QuickFBP algorithm is of polynomial time complexity. Convergence conditions of the QuickFBP, resp. the FBP algorithm are defined and proved for: (1) single output neural networks in case of training patterns with different targets; and (2) multiple output neural networks in case of training patterns with equivalued target vector. They support the automation of the weights training process (quasi-unsupervised learning) establishing the target value(s) depending on the network's input values. In these cases the simulation results confirm the convergence of both algorithms. An example with a large-sized neural network illustrates the significantly greater training speed of the QuickFBP rather than the FBP algorithm. The adaptation of an interactive web system to users on the basis of the QuickFBP algorithm is presented. Since the QuickFBP algorithm ensures quasi-unsupervised learning, this implies its broad applicability in areas of adaptive and adaptable interactive systems, data mining, etc. applications.     

Spatial scene representations formed by self‐organizing learning in a hippocampal extension of the ventral visual system

We show in a unifying computational approach that representations of spatial scenes can be formed by adding an additional self‐organizing layer of processing beyond the inferior temporal visual cortex in the ventral visual stream without the introduction of new computational principles. The invariant representations of objects by neurons in the inferior temporal visual cortex can be modelled by a multilayer feature hierarchy network with feedforward convergence from stage to stage, and an associative learning rule with a short‐term memory trace to capture the invariant statistical properties of objects as they transform over short time periods in the world. If an additional layer is added to this architecture, training now with whole scenes that consist of a set of objects in a given fixed spatial relation to each other results in neurons in the added layer that respond to one of the trained whole scenes but do not respond if the objects in the scene are rearranged to make a new scene from the same objects. The formation of these scene‐specific representations in the added layer is related to the fact that in the inferior temporal cortex and, we show, in the VisNet model, the receptive fields of inferior temporal cortex neurons shrink and become asymmetric when multiple objects are present simultaneously in a natural scene. This reduced size and asymmetry of the receptive fields of inferior temporal cortex neurons also provides a solution to the representation of multiple objects, and their relative spatial positions, in complex natural scenes.     

Accelerating neural network training using weight extrapolations

The backpropagation (BP) algorithm for training feedforward neural networks has proven robust even for difficult problems. However, its high performance results are attained at the expense of a long training time to adjust the network parameters, which can be discouraging in many real-world applications. Even on relatively simple problems, standard BP often requires a lengthy training process in which the complete set of training examples is processed hundreds or thousands of times. In this paper, a universal acceleration technique for the BP algorithm based on extrapolation of each individual interconnection weight is presented. This extrapolation procedure is easy to implement and is activated only a few times in between iterations of the conventional BP algorithm. This procedure, unlike earlier acceleration procedures, minimally alters the computational structure of the BP algorithm. The viability of this new approach is demonstrated on three examples. The results suggest that it leads to significant savings in computation time of the standard BP algorithm. Moreover, the solution computed by the proposed approach is always located in close proximity to the one obtained by the conventional BP procedure. Hence, the proposed method provides a real acceleration of the BP algorithm without degrading the usefulness of its solutions. The performance of the new method is also compared with that of the conjugate gradient algorithm, which is an improved and faster version of the BP algorithm.     

An alternative approach for neural network evolution with a genetic algorithm: crossover by combinatorial optimization.

In this work we present a new approach to crossover operator in the genetic evolution of neural networks. The most widely used evolutionary computation paradigm for neural network evolution is evolutionary programming. This paradigm is usually preferred due to the problems caused by the application of crossover to neural network evolution. However, crossover is the most innovative operator within the field of evolutionary computation. One of the most notorious problems with the application of crossover to neural networks is known as the permutation problem. This problem occurs due to the fact that the same network can be represented in a genetic coding by many different codifications. Our approach modifies the standard crossover operator taking into account the special features of the individuals to be mated. We present a new model for mating individuals that considers the structure of the hidden layer and redefines the crossover operator. As each hidden node represents a non-linear projection of the input variables, we approach the crossover as a problem on combinatorial optimization. We can formulate the problem as the extraction of a subset of near-optimal projections to create the hidden layer of the new network. This new approach is compared to a classical crossover in 25 real-world problems with an excellent performance. Moreover, the networks obtained are much smaller than those obtained with classical crossover operator.     

Dual extended Kalman filtering in recurrent neural networks(1).

In the classical deterministic Elman model, the estimation of parameters must be very accurate. Otherwise, the system performance is very poor. To improve the system performance, we can use a Kalman filtering algorithm to guide the operation of a trained recurrent neural network (RNN). In this case, during training, we need to estimate the state of hidden layer, as well as the weights of the RNN. This paper discusses how to use the dual extended Kalman filtering (DEKF) for this dual estimation and how to use our proposing DEKF for removing some unimportant weights from a trained RNN. In our approach, one Kalman algorithm is used for estimating the state of the hidden layer, and one recursive least square (RLS) algorithm is used for estimating the weights. After training, we use the error covariance matrix of the RLS algorithm to remove unimportant weights. Simulation showed that our approach is an effective joint-learning-pruning method for RNNs under the online operation.     

Logarithmic learning for generalized classifier neural network

Generalized classifier neural network is introduced as an efficient classifier among the others. Unless the initial smoothing parameter value is close to the optimal one, generalized classifier neural network suffers from convergence problem and requires quite a long time to converge. In this work, to overcome this problem, a logarithmic learning approach is proposed. The proposed method uses logarithmic cost function instead of squared error. Minimization of this cost function reduces the number of iterations used for reaching the minima. The proposed method is tested on 15 different data sets and performance of logarithmic learning generalized classifier neural network is compared with that of standard one. Thanks to operation range of radial basis function included by generalized classifier neural network, proposed logarithmic approach and its derivative has continuous values. This makes it possible to adopt the advantage of logarithmic fast convergence by the proposed learning method. Due to fast convergence ability of logarithmic cost function, training time is maximally decreased to 99.2%. In addition to decrease in training time, classification performance may also be improved till 60%. According to the test results, while the proposed method provides a solution for time requirement problem of generalized classifier neural network, it may also improve the classification accuracy. The proposed method can be considered as an efficient way for reducing the time requirement problem of generalized classifier neural network.     

Reorganization of functional brain networks mediates the improvement of cognitive performance following real‐time neurofeedback training of working memory

Working memory (WM) is essential for individuals' cognitive functions. Neuroimaging studies indicated that WM fundamentally relied on a frontoparietal working memory network (WMN) and a cinguloparietal default mode network (DMN). Behavioral training studies demonstrated that the two networks can be modulated by WM training. Different from the behavioral training, our recent study used a real‐time functional MRI (rtfMRI)‐based neurofeedback method to conduct WM training, demonstrating that WM performance can be significantly improved after successfully upregulating the activity of the target region of interest (ROI) in the left dorsolateral prefrontal cortex (Zhang et al., [2013]: PloS One 8:e73735); however, the neural substrate of rtfMRI‐based WM training remains unclear. In this work, we assessed the intranetwork and internetwork connectivity changes of WMN and DMN during the training, and their correlations with the change of brain activity in the target ROI as well as with the improvement of post‐training behavior. Our analysis revealed an “ROI‐network‐behavior” correlation relationship underlying the rtfMRI training. Further mediation analysis indicated that the reorganization of functional brain networks mediated the effect of self‐regulation of the target brain activity on the improvement of cognitive performance following the neurofeedback training. The results of this study enhance our understanding of the neural basis of real‐time neurofeedback and suggest a new direction to improve WM performance by regulating the functional connectivity in the WM related networks. Hum Brain Mapp 36:1705–1715, 2015. © 2014 Wiley Periodicals, Inc.     

Informatic parcellation of the network involved in the computation of subjective value

Understanding how the brain computes value is a basic question in neuroscience. Although individual studies have driven this progress, meta-analyses provide an opportunity to test hypotheses that require large collections of data. We carry out a meta-analysis of a large set of functional magnetic resonance imaging studies of value computation to address several key questions. First, what is the full set of brain areas that reliably correlate with stimulus values when they need to be computed? Second, is this set of areas organized into dissociable functional networks? Third, is a distinct network of regions involved in the computation of stimulus values at decision and outcome? Finally, are different brain areas involved in the computation of stimulus values for different reward modalities? Our results demonstrate the centrality of ventromedial prefrontal cortex (VMPFC), ventral striatum and posterior cingulate cortex (PCC) in the computation of value across tasks, reward modalities and stages of the decision-making process. We also find evidence of distinct subnetworks of co-activation within VMPFC, one involving central VMPFC and dorsal PCC and another involving more anterior VMPFC, left angular gyrus and ventral PCC. Finally, we identify a posterior-to-anterior gradient of value representations corresponding to concrete-to-abstract rewards.     

Training Echo State Networks with Regularization Through Dimensionality Reduction

In this paper, we introduce a new framework to train a class of recurrent neural network, called Echo State Network, to predict real valued time-series and to provide a visualization of the modeled system dynamics. The method consists in projecting the output of the internal layer of the network on a lower dimensional space, before training the output layer to learn the target task. Notably, we enforce a regularization constraint that leads to better generalization capabilities. We evaluate the performances of our approach on several benchmark tests, using different techniques to train the readout of the network, achieving superior predictive performance when using the proposed framework. Finally, we provide an insight on the effectiveness of the implemented mechanics through a visualization of the trajectory in the phase space and relying on the methodologies of nonlinear time-series analysis. By applying our method on well-known chaotic systems, we provide evidence that the lower dimensional embedding retains the dynamical properties of the underlying system better than the full-dimensional internal states of the network.     

A connectionist model for convex-hull of a planar set.

A neural network model is proposed for computation of the convex-hull of a finite planar set. The model is self-organizing in that it adapts itself to the hull-vertices of the convex-hull in an orderly fashion without any supervision. The proposed network consists of three layers of processors. The bottom layer computes the activation functions, the outputs of which are passed onto the middle layer. The middle layer is used for winner selection. These information are passed onto the topmost layer as well as fed back to the bottom layer. The network in the topmost layer self-organizes by labeling the hull-processors in an orderly fashion so that the final convex-hull is obtained from the topmost layer. Time complexities of the proposed model are analyzed and are compared with existing models of similar nature.     

Efficient self-organizing multilayer neural network for nonlinear system modeling

It has been shown extensively that the dynamic behaviors of a neural system are strongly influenced by the network architecture and learning process. To establish an artificial neural network (ANN) with self-organizing architecture and suitable learning algorithm for nonlinear system modeling, an automatic axon–neural network (AANN) is investigated in the following respects. First, the network architecture is constructed automatically to change both the number of hidden neurons and topologies of the neural network during the training process. The approach introduced in adaptive connecting-and-pruning algorithm (ACP) is a type of mixed mode operation, which is equivalent to pruning or adding the connecting of the neurons, as well as inserting some required neurons directly. Secondly, the weights are adjusted, using a feedforward computation (FC) to obtain the information for the gradient during learning computation. Unlike most of the previous studies, AANN is able to self-organize the architecture and weights, and to improve the network performances. Also, the proposed AANN has been tested on a number of benchmark problems, ranging from nonlinear function approximating to nonlinear systems modeling. The experimental results show that AANN can have better performances than that of some existing neural networks.     

Anosognosia and default mode subnetwork dysfunction in Alzheimer's disease

Research on the neural correlates of anosognosia in Alzheimer's disease varied according to methods and objectives: they compared different measures, used diverse neuroimaging modalities, explored connectivity between brain networks, addressed the role of specific brain regions or tried to give support to theoretical models of unawareness. We used resting‐state fMRI connectivity with two different seed regions and two measures of anosognosia in different patient samples to investigate consistent modifications of default mode subnetworks and we aligned the results with the Cognitive Awareness Model. In a first study, patients and their relatives were presented with the Memory Awareness Rating Scale. Anosognosia was measured as a patient‐relative discrepancy score and connectivity was investigated with a parahippocampal seed. In a second study, anosognosia was measured in patients with brain amyloid (taken as a disease biomarker) by comparing self‐reported rating with memory performance, and connectivity was examined with a hippocampal seed. In both studies, anosognosia was consistently related to disconnection within the medial temporal subsystem of the default mode network, subserving episodic memory processes. Importantly, scores were also related to disconnection between the medial temporal and both the core subsystem (participating to self‐reflection) and the dorsomedial subsystem of the default mode network (the middle temporal gyrus that might subserve a personal database in the second study). We suggest that disparity in connectivity within and between subsystems of the default mode network may reflect impaired functioning of pathways in cognitive models of awareness.     

Analysis of the process of visual pattern recognition by the neocognitron

A neural network model of visual pattern recognition called the “neocognitron,” was earlier proposed by the author. It is capable of deformation-invariant visual pattern recognition. After learning, it can recognize input patterns without being affected by deformation, changes in size, or shifts in position. This paper offers a mathematical analysis of the process of visual pattern recognition by the neocognitron. The neocognitron is a hierarchical multilayered network. Its initial stage is an input layer, and each succeeding stage has a layer of “S-cells” followed by a layer of “C-cells.” Thus, in the whole network, layers of S-cells and C-cells are arranged alternately. The process of feature extraction by an S-cell is analyzed mathematically in this paper, and the role of the C-cells in deformation-invariant pattern recognition is discussed.     

Short utterance recognition using a network with minimum training

A feedforward network is used to recognize short, digitized, isolated utterances. A high, multispeaker recognition rate is achieved with a small vocabulary with a single training utterance. This approach makes use of the pattern recognition property of the network architecture to classify different temporal patterns in the multidimensional feature space. The network recognizes the utterances without the need of segmentation, phoneme identification, or time alignment. We train the network with four words spoken by one single speaker. The network is then able to recognize 20 tokens spoken by 5 other speakers. We repeat the above training and testing procedure using a different speaker's utterances for training each time. The overall accuracy is 97.5%. We compare this approach to the traditional dynamic programming (DP) approach, and find that DP with slope constraints of 0 and 1 achieve 98.5% and 85% accuracies respectively. Finally we validate out statistics by training and testing the network of a four-word subset of the Texas Instruments (Tl) isolated word database. The accuracy with this vocabulary exceeds 96%. By doubling the size of the training set, the accuracy is raised to 98%. Using a suitable threshold, we are able to raise the accuracy of one network from 87% to 98.5%. Thresholding applied to all networks would then raise the overall accuracy to well over 99%.

This technique is especially promising because of the low overhead and computational requirements, which make it suitable for a low cost, portable, command recognition type of application.     

An approach to seizure detection using an artificial neural network (ANN)

We have developed an EEG seizure detector based on an artificial neural network. The input layer of the ANN has 31 nodes quantifying the amplitude, slope, curvature, rhythmicity, and frequency components of EEG in a 2 sec epoch. The hidden layer has 30 nodes and the output layer has 8 nodes representing various patterns of EEG activity (e.g. seizure, muscle, noise, normal). The value of the output node representing seizure activity is averaged over 3 consecutive epochs and a seizure is declared when that average exceeds 0.65.Among 78 randomly selected files from 50 patients not in the original training set, the detector declared at least one seizure in 76% of 34 files containing seizures. It declared no seizures in 93% of 44 files not containing seizures. Four false detections during 4.1 h of recording yielded a false detection rate of 1.0/h. The detector can continuously process 40 channels of EEG with a 33 MHz 486 CPU.Although this method is still in its early stages of development, our results represent proof of the principle that ANN could be utilized to provide a practical approach for automatic, on-line, seizure detection.     

The recursive deterministic perceptron neural network

We introduce a feedforward multilayer neural network which is a generalization of the single layer perceptron topology (SLPT), called recursive deterministic perceptron (RDP). This new model is capable of solving any two-class classification problem, as opposed to the single layer perceptron which can only solve classification problems dealing with linearly separable sets (two subsets X and Y of ^d are said to be linearly separable if there exists a hyperplane such that the elements of X and Y lie on the two opposite sides of ^d delimited by this hyperplane). We propose several growing methods for constructing a RDP. These growing methods build a RDP by successively adding intermediate neurons (IN) to the topology (an IN corresponds to a SLPT). Thus, as a result, we obtain a multilayer perceptron topology, which together with the weights, are determined automatically by the constructing algorithms. Each IN augments the affine dimension of the set of input vectors. This augmentation is done by adding the output of each of these INs, as a new component, to every input vector. The construction of a new IN is made by selecting a subset from the set of augmented input vectors which is LS from the rest of this set. This process ends with LS classes in almost n−1 steps where n is the number of input vectors. For this construction, if we assume that the selected LS subsets are of maximum cardinality, the problem is proven to be NP-complete. We also introduce a generalization of the RDP model for classification of m classes (m>2) allowing to always separate m classes. This generalization is based on a new notion of linear separability for m classes, and it follows naturally from the RDP. This new model can be used to compute functions with a finite domain, and thus, to approximate continuous functions. We have also compared — over several classification problems — the percentage of test data correctly classified, or the topology of the 2 and m classes RDPs with that of the backpropagation (BP), cascade correlation (CC), and two other growing methods.