Bayesian neural networks for classification: how useful is the evidence framework?

This paper presents an empirical assessment of the Bayesian evidence framework for neural networks using four synthetic and four real-world classification problems. We focus on three issues; model selection, automatic relevance determination (ARD) and the use of committees. Model selection using the evidence criterion is only tenable if the number of training examples exceeds the number of network weights by a factor of five or ten. With this number of available examples, however, cross-validation is a viable alternative. The ARD feature selection scheme is only useful in networks with many hidden units and for data sets containing many irrelevant variables. ARD is also useful as a hard feature selection method. Results on applying the evidence framework to the real-world data sets showed that committees of Bayesian networks achieved classification accuracies similar to the best alternative methods. Importantly, this was achievable with a minimum of human intervention.     

A closed-form neural network for discriminatory feature extraction from high-dimensional data.

We consider a new neural network for data discrimination in pattern recognition applications. We refer to this as a maximum discriminating feature (MDF) neural network. Its weights are obtained in closed-form, thereby overcoming problems associated with other nonlinear neural networks. It uses neuron activation functions that are dynamically chosen based on the application. It is theoretically shown to provide nonlinear transforms of the input data that are more general than those provided by other nonlinear multilayer perceptron neural network and support-vector machine techniques for cases involving high-dimensional (image) inputs where training data are limited and the classes are not linearly separable. We experimentally verify this on synthetic examples.     

Recognition invariance obtained by extended and invariant features.

In performing recognition, the visual system shows a remarkable capacity to distinguish between significant and immaterial image changes, to learn from examples to recognize new classes of objects, and to generalize from known to novel objects. Here we focus on one aspect of this problem, the ability to recognize novel objects from different viewing directions. This problem of view-invariant recognition is difficult because the image of an object seen from a novel viewing direction can be substantially different from all previously seen images of the same object. We describe an approach to view-invariant recognition that uses extended features to generalize across changes in viewing directions. Extended features are equivalence classes of informative image fragments, which represent object parts under different viewing conditions. This representation is extracted during learning from images of moving objects, and it allows the visual system to generalize from a single view of a novel object, and to compensate for large changes in the viewing direction, without using three-dimensional information. We describe the model, its implementation and performance on natural face images, compare it to alternative approaches, discuss its biological plausibility, and its extension to other aspects of visual recognition. The results of the study suggest that the capacity of the recognition system to generalize to novel conditions in an efficient and flexible manner depends on the ongoing extraction of different families of informative features, acquired for different tasks and different object classes.     

Three learning phases for radial-basis-function networks.

In this paper, learning algorithms for radial basis function (RBF) networks are discussed. Whereas multilayer perceptrons (MLP) are typically trained with backpropagation algorithms, starting the training procedure with a random initialization of the MLP's parameters, an RBF network may be trained in many different ways. We categorize these RBF training methods into one-, two-, and three-phase learning schemes. Two-phase RBF learning is a very common learning scheme. The two layers of an RBF network are learnt separately; first the RBF layer is trained, including the adaptation of centers and scaling parameters, and then the weights of the output layer are adapted. RBF centers may be trained by clustering, vector quantization and classification tree algorithms, and the output layer by supervised learning (through gradient descent or pseudo inverse solution). Results from numerical experiments of RBF classifiers trained by two-phase learning are presented in three completely different pattern recognition applications: (a) the classification of 3D visual objects; (b) the recognition hand-written digits (2D objects); and (c) the categorization of high-resolution electrocardiograms given as a time series (ID objects) and as a set of features extracted from these time series. In these applications, it can be observed that the performance of RBF classifiers trained with two-phase learning can be improved through a third backpropagation-like training phase of the RBF network, adapting the whole set of parameters (RBF centers, scaling parameters, and output layer weights) simultaneously. This, we call three-phase learning in RBF networks. A practical advantage of two- and three-phase learning in RBF networks is the possibility to use unlabeled training data for the first training phase. Support vector (SV) learning in RBF networks is a different learning approach. SV learning can be considered, in this context of learning, as a special type of one-phase learning, where only the output layer weights of the RBF network are calculated, and the RBF centers are restricted to be a subset of the training data. Numerical experiments with several classifier schemes including k-nearest-neighbor, learning vector quantization and RBF classifiers trained through two-phase, three-phase and support vector learning are given. The performance of the RBF classifiers trained through SV learning and three-phase learning are superior to the results of two-phase learning, but SV learning often leads to complex network structures, since the number of support vectors is not a small fraction of the total number of data points.     

Fast Image Recognition Based on Independent Component Analysis and Extreme Learning Machine

Nowadays, image recognition has become a highly active research topic in cognitive computation community, due to its many potential applications. Generally, the image recognition task involves two subtasks: image representation and image classification. Most feature extraction approaches for image representation developed so far regard independent component analysis (ICA) as one of the essential means. However, ICA has been hampered by its extremely expensive computational cost in real-time implementation. To address this problem, a fast cognitive computational scheme for image recognition is presented in this paper, which combines ICA and the extreme learning machine (ELM) algorithm. It tries to solve the image recognition problem at a much faster speed by using ELM not only in image classification but also in feature extraction for image representation. As an example, our proposed approach is applied to the face image recognition with detailed analysis. Firstly, common feature hypothesis is introduced to extract the common visual features from universal images by the traditional ICA model in the offline recognition process, and then ELM is used to simulate ICA for the purpose of facial feature extraction in the online recognition process. Lastly, the resulting independent feature representation of the face images extracted by ELM rather than ICA will be fed into the ELM classifier, which is composed of numerous single hidden layer feed-forward networks. Experimental results on Yale face database and MNIST digit database have shown the good performance of our proposed approach, which could be comparable to the state-of-the-art techniques at a much faster speed.     

Comparison of Feature Selection Techniques in Machine Learning for Anatomical Brain MRI in Dementia

We present a comparative split-half resampling analysis of various data driven feature selection and classification methods for the whole brain voxel-based classification analysis of anatomical magnetic resonance images. We compared support vector machines (SVMs), with or without filter based feature selection, several embedded feature selection methods and stability selection. While comparisons of the accuracy of various classification methods have been reported previously, the variability of the out-of-training sample classification accuracy and the set of selected features due to independent training and test sets have not been previously addressed in a brain imaging context. We studied two classification problems: 1) Alzheimer’s disease (AD) vs. normal control (NC) and 2) mild cognitive impairment (MCI) vs. NC classification. In AD vs. NC classification, the variability in the test accuracy due to the subject sample did not vary between different methods and exceeded the variability due to different classifiers. In MCI vs. NC classification, particularly with a large training set, embedded feature selection methods outperformed SVM-based ones with the difference in the test accuracy exceeding the test accuracy variability due to the subject sample. The filter and embedded methods produced divergent feature patterns for MCI vs. NC classification that suggests the utility of the embedded feature selection for this problem when linked with the good generalization performance. The stability of the feature sets was strongly correlated with the number of features selected, weakly correlated with the stability of classification accuracy, and uncorrelated with the average classification accuracy.     

Position invariant recognition in the visual system with cluttered environments.

The effects of cluttered environments are investigated on the performance of a hierarchical multilayer model of invariant object recognition in the visual system (VisNet) that employs learning rules that utilise a trace of previous neural activity. This class of model relies on the spatio-temporal statistics of natural visual inputs to be able to associate together different exemplars of the same stimulus or object which will tend to occur in temporal proximity. In this paper the different exemplars of a stimulus are the same stimulus in different positions. First it is shown that if the stimuli have been learned previously against a plain background, then the stimuli can be correctly recognised even in environments with cluttered (e.g. natural) backgrounds which form complex scenes. Second it is shown that the functional architecture has difficulty in learning new objects if they are presented against cluttered backgrounds. It is suggested that processes such as the use of a high-resolution fovea, or attention, may be particularly useful in suppressing the effects of background noise and in segmenting objects from their background when new objects need to be learned. However, it is shown third that this problem may be ameliorated by the prior existence of stimulus tuned feature detecting neurons in the early layers of the VisNet, and that these feature detecting neurons may be set up through previous exposure to the relevant class of objects. Fourth we extend these results to partially occluded objects, showing that (in contrast with many artificial vision systems) correct recognition in this class of architecture can occur if the objects have been learned previously without occlusion.     

Boosting feature selection for Neural Network based regression

The head pose estimation problem is well known to be a challenging task in computer vision and is a useful tool for several applications involving human–computer interaction. This problem can be stated as a regression one where the input is an image and the output is pan and tilt angles. Finding the optimal regression is a hard problem because of the high dimensionality of the input (number of image pixels) and the large variety of morphologies and illumination. We propose a new method combining a boosting strategy for feature selection and a neural network for the regression. Potential features are a very large set of Haar-like wavelets which are well known to be adapted to face image processing. To achieve the feature selection, a new Fuzzy Functional Criterion (FFC) is introduced which is able to evaluate the link between a feature and the output without any estimation of the joint probability density function as in the Mutual Information. The boosting strategy uses this criterion at each step: features are evaluated by the FFC using weights on examples computed from the error produced by the neural network trained at the previous step. Tests are carried out on the commonly used Pointing 04 database and compared with three state-of-the-art methods. We also evaluate the accuracy of the estimation on FacePix, a database with a high angular resolution. Our method is compared positively to a Convolutional Neural Network, which is well known to incorporate feature extraction in its first layers.     

Multiple elastic modules for visual pattern recognition

Fast synaptic plasticity, used to associate topologically ordered features in an input image to those of previously learned objects, has been previously proposed as a possible model for object recognition (von der Malsburg & Bienenstock, 1987, Europhysics Letters, 3(11), 1243–1249). In this paper, it is argued that in addition to rapid link dynamics, fast receptive field size dynamics are necessary to automatically escape from poor local matches and also allow for simultaneous recognition of multiple objects. Furthermore, a feature locking mechanism with a properly designed hysteresis property is needed to handle complex, cluttered, and dynamic scenes. The multiple elastic modules (MEM) model, described in this paper, utilizes newly developed dynamics that locate and recognize a previously learned object based on expected spatial arrangement of local features. The MEM model can be viewed as using a deformable template of an object to search the input scene. Unlike many of the current artificial neural network models, the proposed MEM model attempts to capture many of the functions available in the biological visual system by providing mechanisms for: multi-model feature integration, generation and maintenance of focus of attention, multiresolution hierarchical searching, and top-down expectation driven processing coupled with bottom-up feature activation processing. In addition, the MEM dynamics, unlike similar template matching approaches (Konen et al., 1994, Neural Networks, 7(6/7), 1019–1030; Yuille et al., 1992, International Journal of Computer Vision, 8(2), 99–111), does not converge to false objects when there are no sufficiently familiar objects in the scene. The performance of the MEM model in detection and recognition of objects through a number of computer simulations is demonstrated.     

Learning invariant object recognition from temporal correlation in a hierarchical network

Invariant object recognition, which means the recognition of object categories independent of conditions like viewing angle, scale and illumination, is a task of great interest that humans can fulfill much better than artificial systems. During the last years several basic principles were derived from neurophysiological observations and careful consideration: (1) Developing invariance to possible transformations of the object by learning temporal sequences of visual features that occur during the respective alterations. (2) Learning in a hierarchical structure, so basic level (visual) knowledge can be reused for different kinds of objects. (3) Using feedback to compare predicted input with the current one for choosing an interpretation in the case of ambiguous signals. In this paper we propose a network which implements all of these concepts in a computationally efficient manner which gives very good results on standard object datasets. By dynamically switching off weakly active neurons and pruning weights computation is sped up and thus handling of large databases with several thousands of images and a number of categories in a similar order becomes possible. The involved parameters allow flexible adaptation to the information content of training data and allow tuning to different databases relatively easily. Precondition for successful learning is that training images are presented in an order assuring that images of the same object under similar viewing conditions follow each other. Through an implementation with sparse data structures the system has moderate memory demands and still yields very good recognition rates.     

An empirical comparison of back propagation and the RDSE algorithm on continuously valued real world data

The ability of a neural network to generalise is dependent on how representative the training patterns were of the whole data domain, and how smoothly the network has fitted to these patterns [Sethi, I.K. (1990). IEEE International Joint Conference on Neural Networks, Seattle, WA, Vol. 2, pp. 219–224]. In non-scaled continuous data domains, training examples will lie at differing distances from each other, making the fitting problem more difficult and varied. This paper introduces a new neuron with an adaptive steepness parameter, implemented as an extra internal connection, which is altered to better interpolate between the data points that its hyperplane divides. Networks of the new neuronal model are trained using a new paradigm entitled the random directed search by entropy algorithm (RDSE). This involves constructing a network by training one neuron at a time and freezing the weights. Each neuron is trained using directed random search [Baba (1989). Neural Networks, 2, 367–373] to find a hyperplane that separates examples by minimising an entropy measure [Quinlan (1986). Induction of Decision Trees, Machine Learning, Vol. 1, pp. 81–106]. This training paradigm solves the problem of pre-defining a network topology, has few problems with local minima, can handle unscaled continuous input data and can be fully trained in a relatively short time scale when compared with other methods, e.g. back propagation (BP).

An example benchmark problem is used to illustrate the effects of the new neuronal model, and results for two real world data domains are given which display an improved classification rate when compared against networks with a constant steepness value for every neuron. An empirical comparison between BP and RDSE for the two data sets are also given. These results display improved training times, robustness and classification rates by RDSE when compared against BP.     

Quantum inspired PSO for the optimization of simultaneous recurrent neural networks as MIMO learning systems

Training a single simultaneous recurrent neural network (SRN) to learn all outputs of a multiple-input–multiple-output (MIMO) system is a difficult problem. A new training algorithm developed from combined concepts of swarm intelligence and quantum principles is presented. The training algorithm is called particle swarm optimization with quantum infusion (PSO-QI). To improve the effectiveness of learning, a two-step learning approach is introduced in the training. The objective of the learning in the first step is to find the optimal set of weights in the SRN considering all output errors. In the second step, the objective is to maximize the learning of each output dynamics by fine tuning the respective SRN output weights. To demonstrate the effectiveness of the PSO-QI training algorithm and the two-step learning approach, two examples of an SRN learning MIMO systems are presented. The first example is learning a benchmark MIMO system and the second one is the design of a wide area monitoring system for a multimachine power system. From the results, it is observed that SRNs can effectively learn MIMO systems when trained using the PSO-QI algorithm and the two-step learning approach.     

On the capabilities of neural networks using limited precision weights.

This paper analyzes some aspects of the computational power of neural networks using integer weights in a very restricted range. Using limited range integer values opens the road for efficient VLSI implementations because: (i) a limited range for the weights can be translated into reduced storage requirements and (ii) integer computation can be implemented in a more efficient way than the floating point one. The paper concentrates on classification problems and shows that, if the weights are restricted in a drastic way (both range and precision), the existence of a solution is not to be taken for granted anymore. The paper presents an existence result which relates the difficulty of the problem as characterized by the minimum distance between patterns of different classes to the weight range necessary to ensure that a solution exists. This result allows us to calculate a weight range for a given category of problems and be confident that the network has the capability to solve the given problems with integer weights in that range. Worst-case lower bounds are given for the number of entropy bits and weights necessary to solve a given problem. Various practical issues such as the relationship between the information entropy bits and storage bits are also discussed. The approach presented here uses a worst-case analysis. Therefore, the approach tends to overestimate the values obtained for the weight range, the number of bits and the number of weights. The paper also presents some statistical considerations that can be used to give up the absolute confidence of a successful training in exchange for values more appropriate for practical use. The approach presented is also discussed in the context of the VC-complexity.     

Oriented principal component analysis for large margin classifiers

Large margin classifiers (such as MLPs) are designed to assign training samples with high confidence (or margin) to one of the classes. Recent theoretical results of these systems show why the use of regularisation terms and feature extractor techniques can enhance their generalisation properties. Since the optimal subset of features selected depends on the classification problem, but also on the particular classifier with which they are used, global learning algorithms for large margin classifiers that use feature extractor techniques are desired. A direct approach is to optimise a cost function based on the margin error, which also incorporates regularisation terms for controlling capacity. These terms must penalise a classifier with the largest margin for the problem at hand. Our work shows that the inclusion of a PCA term can be employed for this purpose. Since PCA only achieves an optimal discriminatory projection for some particular distribution of data, the margin of the classifier can then be effectively controlled. We also propose a simple constrained search for the global algorithm in which the feature extractor and the classifier are trained separately. This allows a degree of flexibility for including heuristics that can enhance the search and the performance of the computed solution. Experimental results demonstrate the potential of the proposed method.     

Feature Component-Based Extreme Learning Machines for Finger Vein Recognition

This paper proposes an efficient finger vein recognition system, in which a variant of the original ensemble extreme learning machine (ELM) called the feature component-based ELMs (FC-ELMs) designed to utilize the characteristics of the features, is introduced to improve the recognition accuracy and stability and to substantially reduce the number of hidden nodes. For feature extraction, an explicit guided filter is proposed to extract the eight block-based directional features from the high-quality finger vein contours obtained from noisy, non-uniform, low-contrast finger vein images without introducing any segmentation process. An FC-ELMs consist of eight single ELMs, each trained with a block feature with a pre-defined direction to enhance the robustness against variation of the finger vein images, and an output layer to combine the outputs of the eight ELMs. For the structured training of the vein patterns, the FC-ELMs are designed to first train small differences between patterns with the same angle and then to aggregate the differences at the output layer. Each ELM can easily learn lower-complexity patterns with a smaller network and the matching accuracy can also be improved, due to the less complex boundaries required for each ELM. We also designed the ensemble FC-ELMs to provide the matching system with stability. For the dataset considered, the experimental results show that the proposed system is able to generate clearer vein contours and has good matching performance with an accuracy of 99.53 % and speed of 0.87 ms per image.     

How to compare two quantities? A computational model of flutter discrimination

A task that has been intensively studied at the neural level is f lutter discrimination. I argue that f lutter discrimination entails a combination of a temporal assignment problem and a quantity comparison problem, and propose a neural network model of how these problems are solved. The network combines unsupervised and one-layer supervised training. The unsupervised part clusters input features (stimulus + time window) and the supervised part categorizes the resulting clusters. After training, the model shows a good fit with both neural and behavioral properties. New predictions are outlined and links with other cognitive domains are pointed out.     

Learning using privileged information: SVM+ and weighted SVM

Prior knowledge can be used to improve predictive performance of learning algorithms or reduce the amount of data required for training. The same goal is pursued within the learning using privileged information paradigm which was recently introduced by Vapnik et al. and is aimed at utilizing additional information available only at training time—a framework implemented by SVM+. We relate the privileged information to importance weighting and show that the prior knowledge expressible with privileged features can also be encoded by weights associated with every training example. We show that a weighted SVM can always replicate an SVM+ solution, while the converse is not true and we construct a counterexample highlighting the limitations of SVM+. Finally, we touch on the problem of choosing weights for weighted SVMs when privileged features are not available.     

Entwicklung interpersoneller Problemlösekompetenz während psychotherapeutischer Ausbildung in Psychodrama

Psychotherapists’ skills regarding relationships are the key elements of successful psychotherapy for all psychotherapeutic schools. During psychotherapy training programs it is essential to train those skills. This survey aims of finding out if trainees learn relational and interpersonal problem solving competencies during their psychotherapy training and if these competencies are measurable and evaluable. For this reason, the psychodrama training program at the “Institute for Psychosocial Intervention and Communications Research” (University of Innsbruck) was subject of research over the past years. Besides other instruments, the IIP—Inventory of Interpersonal Problems—has been used to evaluate the training program. This pilot survey shows that the IIP is appropriate for therapist training studies. During the psychodrama training program main features of relational competencies have improved significantly. The trainees’ interpersonal problem solving competencies have clearly increased.