A recurrent neural network: Limitations and training

A recurrent, synchronous neural network is treated as a collection of independent perceptrons. The dynamics of the network can be described by a mapping: a finite set of transitions in the state space of the network. We define legal mapping as a mapping that a synchronous neural network is able to perform, and state the necessary and sufficient conditions for a mapping to be legal. A learning algorithm for the network, based on the perceptron's learning algorithm, is guaranteed to converge to a solution when the network is trained to realize a legal mapping. It is shown that the algorithm performs a gradient descent search for a minimum of a cost function that is a certain error measure in the weight space. Performance of the algorithm for the associative memory application and for temporal sequences production is illustrated by numerical simulations. A method is proposed for legalizing any given mapping at the expense of adding a finite number of neurons to the network. It is also shown that when the number of transitions in a random mapping is less than the number of neurons in the network, the probability that such a mapping is legal approaches unity.     

Epilepsy classification using optimized artificial neural network

ABSTRACT

Objectives: An Electroencephalogram (EEG) is the result of co-operative actions performed by brain cells. In other words, it can be defined as the time course of extracellular field potentials that are generated due to the synchronous action of cells. It is widely used for the analysis and diagnosis of several conditions. But this clinical data use to be multi-dimensional, context-dependent, complex, and it causes a concoction of various computing related research challenges. The objective of this study was to develop a computer-aided diagnosis system for epilepsy detection using EEG signals to ease the diagnosis process.

Materials: In this study, EEG datasets for epilepsy disease detection were taken from a public domain (Bonn University, Germany). These EEG recordings contain 100 single-channel EEG signals with maximum duration of 23.6 seconds. This data set was recorded intra-cranially and extra-cranially with the help of a 128-channel amplifier system using a common reference point.

Results: For a unique set of EEG signal features, the Optimized Artificial Neural Network model for classification and validation was developed with optimum neurons in the hidden layer. Results were tested on the basis of accuracy, sensitivity, precision, and specificity for all classes. The proposed Particle Swarm Optimized Artificial Neural Network provided 99.3% accuracy for EEG signal classification.

Discussion: Our results indicate that artificial neural network has efficiency to provide higher accuracy for epilepsy detection if the statistical features are extracted carefully. It is also possible to improve results for real time diagnosis by using optimization technique for error reduction.

Abbreviations: EEG: Electroencephalogram CAD: Computer-Aided Diagnosis ANN: Artificial Neural Network PSO: Particle Swarm Optimization FIR: Finite Impulse Response IIR: Infinite Impulse Response MSE: Mean Square Error.     

Classification of EMG signals using wavelet neural network

An accurate and computationally efficient means of classifying electromyographic (EMG) signal patterns has been the subject of considerable research effort in recent years. Quantitative analysis of EMG signals provides an important source of information for the diagnosis of neuromuscular disorders. Following the recent development of computer-aided EMG equipment, different methodologies in the time domain and frequency domain have been followed for quantitative analysis. In this study, feedforward error backpropagation artificial neural networks (FEBANN) and wavelet neural networks (WNN) based classifiers were developed and compared in relation to their accuracy in classification of EMG signals. In these methods, we used an autoregressive (AR) model of EMG signals as an input to classification system. A total of 1200 MUPs obtained from 7 normal subjects, 7 subjects suffering from myopathy and 13 subjects suffering from neurogenic disease were analyzed. The success rate for the WNN technique was 90.7% and for the FEBANN technique 88%. The comparisons between the developed classifiers were primarily based on a number of scalar performance measures pertaining to the classification. The WNN-based classifier outperformed the FEBANN counterpart. The proposed WNN classification may support expert decisions and add weight to EMG differential diagnosis.     

Relation between weight size and degree of over-fitting in neural network regression

This paper investigates the relation between over-fitting and weight size in neural network regression. The over-fitting of a network to Gaussian noise is discussed. Using re-parametrization, a network function is represented as a bounded function g multiplied by a coefficient c. This is considered to bound the squared sum of the outputs of g at given inputs away from a positive constant delta(n), which restricts the weight size of a network and enables the probabilistic upper bound of the degree of over-fitting to be derived. This reveals that the order of the probabilistic upper bound can change depending on delta(n). By applying the bound to analyze the over-fitting behavior of one Gaussian unit, it is shown that the probability of obtaining an extremely small value for the width parameter in training is close to one when the sample size is large.     

Online data visualization using the neural gas network.

A high-quality distance preserving output representation is provided to the neural gas (NG) network. The nonlinear mapping is determined concurrently along with the codebook vectors. The adaptation rule for codebook positions in the projection space minimizes a cost function that favors the trustworthy preservation of the local topology. The proposed visualization method, called OVI-NG, is an enhancement over curvilinear component analysis (CCA). The results show that the mapping quality obtained with OVI-NG outperforms the original CCA, in terms of the trustworthiness, continuity, topographic function and topology preservation measures.     

Short utterance recognition using a network with minimum training

A feedforward network is used to recognize short, digitized, isolated utterances. A high, multispeaker recognition rate is achieved with a small vocabulary with a single training utterance. This approach makes use of the pattern recognition property of the network architecture to classify different temporal patterns in the multidimensional feature space. The network recognizes the utterances without the need of segmentation, phoneme identification, or time alignment. We train the network with four words spoken by one single speaker. The network is then able to recognize 20 tokens spoken by 5 other speakers. We repeat the above training and testing procedure using a different speaker's utterances for training each time. The overall accuracy is 97.5%. We compare this approach to the traditional dynamic programming (DP) approach, and find that DP with slope constraints of 0 and 1 achieve 98.5% and 85% accuracies respectively. Finally we validate out statistics by training and testing the network of a four-word subset of the Texas Instruments (Tl) isolated word database. The accuracy with this vocabulary exceeds 96%. By doubling the size of the training set, the accuracy is raised to 98%. Using a suitable threshold, we are able to raise the accuracy of one network from 87% to 98.5%. Thresholding applied to all networks would then raise the overall accuracy to well over 99%.

This technique is especially promising because of the low overhead and computational requirements, which make it suitable for a low cost, portable, command recognition type of application.     

Fast robust subject-independent magnetoencephalographic source localization using an artificial neural network

We describe a system that localizes a single dipole to reasonable accuracy from noisy magnetoencephalographic (MEG) measurements in real time. At its core is a multilayer perceptron (MLP) trained to map sensor signals and head position to dipole location. Including head position overcomes the previous need to retrain the MLP for each subject and session. The training dataset was generated by mapping randomly chosen dipoles and head positions through an analytic model and adding noise from real MEG recordings. After training, a localization took 0.7 ms with an average error of 0.90 cm. A few iterations of a Levenberg-Marquardt routine using the MLP output as its initial guess took 15 ms and improved accuracy to 0.53 cm, which approaches the natural limit on accuracy imposed by noise. We applied these methods to localize single dipole sources from MEG components isolated by blind source separation and compared the estimated locations to those generated by standard manually assisted commercial software.     

Classification of clustered microcalcifications using a Shape Cognitron neural network.

A new shape recognition-based neural network built with universal feature planes, called Shape Cognitron (S-Cognitron) is introduced to classify clustered microcalcifications. The architecture of S-Cognitron consists of two modules and an extra layer, called 3D figure layer lies in between. The first module contains a shape orientation layer, built with 20 cell planes of low level universal shape features to convert first-order shape orientations into numeric values, and a complex layer, to extract second-order shape features. The 3D figure layer is a feature extract-display layer that extracts the shape curvatures of an input pattern and displays them as a 3D figure. It is then followed by a second module made up of a feature formation layer and a probabilistic neural network-based classification layer. The system is evaluated by using Nijmegen mammogram database and experimental results show that sensitivity and specificity can reach 86.1 and 74.1%, respectively.     

An empirical evaluation of Bayesian sampling with hybrid Monte Carlo for training neural network classifiers

This article gives a concise overview of Bayesian sampling for neural networks, and then presents an extensive evaluation on a set of various benchmark classification problems. The main objective is to study the sensitivity of this scheme to changes in the prior distribution of the parameters and hyperparameters, and to evaluate the efficiency of the so-called automatic relevance determination (ARD) method. The article concludes with a comparison of the achieved classification results with those obtained with (i) the evidence scheme and (ii) with non-Bayesian methods.     

Generalized classifier neural network

In this work a new radial basis function based classification neural network named as generalized classifier neural network, is proposed.The proposed generalized classifier neural network has five layers, unlike other radial basis function based neural networks such as generalized regression neural network and probabilistic neural network. They are input, pattern, summation, normalization and output layers. In addition to topological difference, the proposed neural network has gradient descent based optimization of smoothing parameter approach and diverge effect term added calculation improvements. Diverge effect term is an improvement on summation layer calculation to supply additional separation ability and flexibility. Performance of generalized classifier neural network is compared with that of the probabilistic neural network, multilayer perceptron algorithm and radial basis function neural network on 9 different data sets and with that of generalized regression neural network on 3 different data sets include only two classes in MATLAB environment. Better classification performance up to %89 is observed. Improved classification performances proved the effectivity of the proposed neural network.     

Predicting ventriculoperitoneal shunt infection in children with hydrocephalus using artificial neural network

Objectives

The relationships between shunt infection and predictive factors have not been previously investigated using Artificial Neural Network (ANN) model. The aim of this study was to develop an ANN model to predict shunt infection in a group of children with shunted hydrocephalus.

Materials and methods

Among more than 800 ventriculoperitoneal shunt procedures which had been performed between April 2000 and April 2011, 68 patients with shunt infection and 80 controls that fulfilled a set of meticulous inclusion/exclusion criteria were consecutively enrolled. Univariate analysis was performed for a long list of risk factors, and those with p value < 0.2 were used to create ANN and logistic regression (LR) models.

Results

Five variables including birth weight, age at the first shunting, shunt revision, prematurity, and myelomeningocele were significantly associated with shunt infection via univariate analysis, and two other variables (intraventricular hemorrhage and coincided infections) had a p value of less than 0.2. Using these seven input variables, ANN and LR models predicted shunt infection with an accuracy of 83.1 % (AUC; 91.98 %, 95 % CI) and 55.7 % (AUC; 76.5, 95 % CI), respectively. The contribution of the factors in the predictive performance of ANN in descending order was history of shunt revision, low birth weight (under 2000 g), history of prematurity, the age at the first shunt procedure, history of intraventricular hemorrhage, history of myelomeningocele, and coinfection.

Conclusion

The findings show that artificial neural networks can predict shunt infection with a high level of accuracy in children with shunted hydrocephalus. Also, the contribution of different risk factors in the prediction of shunt infection can be determined using the trained network.

     

Automatic seizure detection in EEG using logistic regression and artificial neural network

The detection of epileptiform discharges in the EEG is an important component in the diagnosis of epilepsy. In this study, multiple signal classification (MUSIC), autoregressive (AR) and periodogram methods were used to get power spectra in patients with absence seizure. The EEG power spectra were used as an input to a classifier. We introduce two fundamentally different approaches for designing classification models (classifiers); the traditional statistical method based on logistic regression (LR) and the emerging computationally powerful techniques based on artificial neural networks (ANNs). LR as well as multilayer perceptron neural network (MLPNN) based classifiers were developed and compared in relation to their accuracy in classification of EEG signals. The comparisons between the developed classifiers were primarily based on analysis of the receiver operating characteristic (ROC) curves as well as a number of scalar performance measures pertaining to the classification. The MLPNN based classifier outperformed the LR based counterpart. Within the same group, the MLPNN-based classifier was more accurate than the LR-based classifier.     

Upper bound of the expected training error of neural network regression for a Gaussian noise sequence

In neural network regression problems, often referred to as additive noise models, NIC (Network Information Criterion) has been proposed as a general model selection criterion to determine the optimal network size with high generalization performance. Although NIC has been derived using asymptotic expansion, it has been pointed out that this technique cannot be applied under the assumption that a target function is in a family of assumed networks and the family is not minimal for representing the target true function, i.e. the overrealizable case, in which NIC reduces to the well-known AIC (Akaike Information Criterion) and others depending on a loss function. Because NIC is the unbiased estimator of generalization error based on training error, it is required to derive the expectations of errors for neural networks for such cases. This paper gives upper bounds of the expectations of training errors with respect to the distribution of training data, which we call the expected training error, for some types of networks under the squared error loss. In the overrealizable case, because the errors are determined by fitting properties of networks to noise components, including in data, the target set of data is taken to be a Gaussian noise sequence. For radial basis function networks and 3-layered neural networks with bell shaped activation function in the hidden layer, the expected training error is bounded above by σ_*²−2nσ_*²logT/T, where σ_*² is the variance of noise, n is the number of basis functions or the number of hidden units and T is the number of data. Furthermore, for 3-layered neural networks with sigmoidal activation function in the hidden layer, we obtained the upper bound of σ_*²−O(log T/T) when n>2. If the number of data is large enough, these bounds of the expected training error are smaller than σ_*²−N(n)σ_*²/T as evaluated in NIC, where N(n) is the number of all network parameters.     

Local coupled feedforward neural network

In this paper, the local coupled feedforward neural network is presented. Its connection structure is same as that of Multilayer Perceptron with one hidden layer. In the local coupled feedforward neural network, each hidden node is assigned an address in an input space, and each input activates only the hidden nodes near it. For each input, only the activated hidden nodes take part in forward and backward propagation processes. Theoretical analysis and simulation results show that this neural network owns the “universal approximation” property and can solve the learning problem of feedforward neural networks. In addition, its characteristic of local coupling makes knowledge accumulation possible.     

Automatic recognition of alertness level by using wavelet transform and artificial neural network

We propose a novel method for automatic recognition of alertness level from full spectrum electroencephalogram (EEG) recordings. This procedure uses power spectral density (PSD) of discrete wavelet transform (DWT) of full spectrum EEG as an input to an artificial neural network (ANN) with three discrete outputs: alert, drowsy and sleep. The error back propagation neural network is selected as a classifier to discriminate the alertness level of a subject. EEG signals were obtained from 30 healthy subjects. The group consisted of 14 females and 16 males with ages ranging from 18 to 65 years and a mean age of 33.5 years, and a body mass index (BMI) of 32.4 +/- 7.3 kg/m2. Alertness level and classification properties of ANN were tested using the data recorded in 12 healthy subjects, whereby the EEG recordings were not used been used to train the ANN. The statistics were used as a measure of potential applicability of the ANN. The accuracy of the ANN was 96 +/- 3% alert, 95 +/- 4% drowsy and 94 +/- 5% sleep. The results suggest that the automatic recognition algorithm is applicable for distinguishing between alert, drowsy and sleep state in recordings that have not been used for the training.     

Signal decoding and receiver evolution. An analysis using an artificial neural network

We use a connectionist model, a recurrent artificial neural network, to investigate the evolution of species recognition in sympatric taxa. We addressed three questions: (1) Does the accuracy of artificial neural networks in discriminating between conspecifics and other sympatric heterospecifics depend on whether the networks were trained only to recognize conspecifics, as opposed to being trained to discriminate between conspecifics and sympatric heterospecifics? (2) Do artificial neural networks weight most heavily those signal features that differ most between conspecifics and sympatric heterospecifics, or those features that vary less within conspecifics? (3) Does selection for species recognition generate sexual selection? We find that: (1) Neural networks trained only on self recognition do not classify species as accurately as networks trained to discriminate between conspecifics and heterospecifics. (2) Neural networks weight signal features in a manner suggesting that the total sound environment as opposed to the relative variation of signals within the species is more important in the evolution of recognition mechanisms. (3) Selection for species recognition generates substantial variation in the relative attractiveness of signals within the species and thus can result in sexual selection.     

An approach to seizure detection using an artificial neural network (ANN)

We have developed an EEG seizure detector based on an artificial neural network. The input layer of the ANN has 31 nodes quantifying the amplitude, slope, curvature, rhythmicity, and frequency components of EEG in a 2 sec epoch. The hidden layer has 30 nodes and the output layer has 8 nodes representing various patterns of EEG activity (e.g. seizure, muscle, noise, normal). The value of the output node representing seizure activity is averaged over 3 consecutive epochs and a seizure is declared when that average exceeds 0.65.Among 78 randomly selected files from 50 patients not in the original training set, the detector declared at least one seizure in 76% of 34 files containing seizures. It declared no seizures in 93% of 44 files not containing seizures. Four false detections during 4.1 h of recording yielded a false detection rate of 1.0/h. The detector can continuously process 40 channels of EEG with a 33 MHz 486 CPU.Although this method is still in its early stages of development, our results represent proof of the principle that ANN could be utilized to provide a practical approach for automatic, on-line, seizure detection.     

A novel nonlinear adaptive filter using a pipelined second-order Volterra recurrent neural network

To enhance the performance and overcome the heavy computational complexity of recurrent neural networks (RNN), a novel nonlinear adaptive filter based on a pipelined second-order Volterra recurrent neural network (PSOVRNN) is proposed in this paper. A modified real-time recurrent learning (RTRL) algorithm of the proposed filter is derived in much more detail. The PSOVRNN comprises of a number of simple small-scale second-order Volterra recurrent neural network (SOVRNN) modules. In contrast to the standard RNN, these modules of a PSOVRNN can be performed simultaneously in a pipelined parallelism fashion, which can lead to a significant improvement in its total computational efficiency. Moreover, since each module of the PSOVRNN is a SOVRNN in which nonlinearity is introduced by the recursive second-order Volterra (RSOV) expansion, its performance can be further improved. Computer simulations have demonstrated that the PSOVRNN performs better than the pipelined recurrent neural network (PRNN) and RNN for nonlinear colored signals prediction and nonlinear channel equalization. However, the superiority of the PSOVRNN over the PRNN is at the cost of increasing computational complexity due to the introduced nonlinear expansion of each module.