Simultaneous Feature Selection and Support Vector Machine Optimization Using the Grasshopper Optimization Algorithm

Support vector machine (SVM) is considered to be one of the most powerful learning algorithms and is used for a wide range of real-world applications. The efficiency of SVM algorithm and its performance mainly depends on the kernel type and its parameters. Furthermore, the feature subset selection that is used to train the SVM model is another important factor that has a major influence on it classification accuracy. The feature subset selection is a very important step in machine learning, specially when dealing with high-dimensional data sets. Most of the previous researches handled these important factors separately. In this paper, we propose a hybrid approach based on the Grasshopper optimisation algorithm (GOA), which is a recent algorithm inspired by the biological behavior shown in swarms of grasshoppers. The goal of the proposed approach is to optimize the parameters of the SVM model, and locate the best features subset simultaneously. Eighteen low- and high-dimensional benchmark data sets are used to evaluate the accuracy of the proposed approach. For verification, the proposed approach is compared with seven well-regarded algorithms. Furthermore, the proposed approach is compared with grid search, which is the most popular technique for tuning SVM parameters. The experimental results show that the proposed approach outperforms all of the other techniques in most of the data sets in terms of classification accuracy, while minimizing the number of selected features.     

Sleep-stage scoring in the rat using a support vector machine

Analysis and classification of sleep stages is a fundamental part of basic sleep research. Rat sleep stages are scored based on electrocorticographic (ECoG) signals recorded from electrodes implanted epidurally and electromyographic (EMG) signals from the temporalis or nuchal muscle. An automated sleep scoring system was developed using a support vector machine (SVM) to discriminate among waking, nonrapid eye movement sleep, and paradoxical sleep. Two experts scored retrospective data obtained from six Sprague-Dawley rodents to provide the training sets and subsequent comparison data used to assess the effectiveness of the SVM. Numerous time-domain and frequency-domain features were extracted for each epoch and selectively reduced using statistical analyses. The SVM kernel function was chosen to be a Gaussian radial basis function and kernel parameters were varied to examine the effectiveness of optimization methods. Tests indicated that a common set of features could be chosen resulted in an overall agreement between the automated scores and the expert consensus of greater than 96%.     

Semi-supervised learning for ordinal Kernel Discriminant Analysis

Ordinal classification considers those classification problems where the labels of the variable to predict follow a given order. Naturally, labelled data is scarce or difficult to obtain in this type of problems because, in many cases, ordinal labels are given by a user or expert (e.g. in recommendation systems). Firstly, this paper develops a new strategy for ordinal classification where both labelled and unlabelled data are used in the model construction step (a scheme which is referred to as semi-supervised learning). More specifically, the ordinal version of kernel discriminant learning is extended for this setting considering the neighbourhood information of unlabelled data, which is proposed to be computed in the feature space induced by the kernel function. Secondly, a new method for semi-supervised kernel learning is devised in the context of ordinal classification, which is combined with our developed classification strategy to optimise the kernel parameters. The experiments conducted compare 6 different approaches for semi-supervised learning in the context of ordinal classification in a battery of 30 datasets, showing (1) the good synergy of the ordinal version of discriminant analysis and the use of unlabelled data and (2) the advantage of computing distances in the feature space induced by the kernel function.     

An accelerated procedure for recursive feature ranking on microarray data.

We describe a new wrapper algorithm for fast feature ranking in classification problems. The Entropy-based Recursive Feature Elimination (E-RFE) method eliminates chunks of uninteresting features according to the entropy of the weights distribution of a SVM classifier. With specific regard to DNA microarray datasets, the method is designed to support computationally intensive model selection in classification problems in which the number of features is much larger than the number of samples. We test E-RFE on synthetic and real data sets, comparing it with other SVM-based methods. The speed-up obtained with E-RFE supports predictive modeling on high dimensional microarray data.     

Kernel learning at the first level of inference

Kernel learning methods, whether Bayesian or frequentist, typically involve multiple levels of inference, with the coefficients of the kernel expansion being determined at the first level and the kernel and regularisation parameters carefully tuned at the second level, a process known as model selection. Model selection for kernel machines is commonly performed via optimisation of a suitable model selection criterion, often based on cross-validation or theoretical performance bounds. However, if there are a large number of kernel parameters, as for instance in the case of automatic relevance determination (ARD), there is a substantial risk of over-fitting the model selection criterion, resulting in poor generalisation performance. In this paper we investigate the possibility of learning the kernel, for the Least-Squares Support Vector Machine (LS-SVM) classifier, at the first level of inference, i.e. parameter optimisation. The kernel parameters and the coefficients of the kernel expansion are jointly optimised at the first level of inference, minimising a training criterion with an additional regularisation term acting on the kernel parameters. The key advantage of this approach is that the values of only two regularisation parameters need be determined in model selection, substantially alleviating the problem of over-fitting the model selection criterion. The benefits of this approach are demonstrated using a suite of synthetic and real-world binary classification benchmark problems, where kernel learning at the first level of inference is shown to be statistically superior to the conventional approach, improves on our previous work (Cawley and Talbot, 2007) and is competitive with Multiple Kernel Learning approaches, but with reduced computational expense.     

Terminated Ramp–Support Vector Machines: A nonparametric data dependent kernel

We propose a novel algorithm, Terminated Ramp–Support Vector Machines (TR–SVM), for classification and feature ranking purposes in the family of Support Vector Machines. The main improvement relies on the fact that the kernel is automatically determined by the training examples. It is built as a function of simple classifiers, generalized terminated ramp functions, obtained by separating oppositely labeled pairs of training points. The algorithm has a meaningful geometrical interpretation, and it is derived in the framework of Tikhonov regularization theory. Its unique free parameter is the regularization one, representing a trade-off between empirical error and solution complexity. Employing the equivalence between the proposed algorithm and two-layer networks, a theoretical bound on the generalization error is also derived, together with Vapnik–Chervonenkis dimension. Performances are tested on a number of synthetic and real data sets.     

Optimization of wavelets for classification of movement-related cortical potentials generated by variation of force-related parameters

The paper presents a novel pattern recognition approach for the classification of single-trial movement-related cortical potentials (MRCPs) generated by variations of force-related parameters during voluntary tasks. The feature space was built from the coefficients of a discrete dyadic wavelet transformation. Mother wavelet parameterization allowed the tuning of basis functions to project the signals. The mother wavelet was optimized to minimize the classification error estimated from the training set. Classification was performed with a support vector machine (SVM) approach with optimization of the width of a Gaussian kernel and of the regularization parameter. The efficacy of the optimization procedures was representatively shown on electroencephalographic recordings from two subjects who performed unilateral isometric plantar flexions at two target torques and two rates of torque development. The proposed classification method was tested on four pairs of classes corresponding to the change in only one of the two parameters of the task. Misclassification rate (test set) in the classification of 1-s EEG activity immediately before the onset of the tasks was reduced from 50.8+/-2.9% with worst wavelet and nearest representative classifier, to 40.2+/-7.3% with optimal wavelet and nearest representative classifier, and to 15.8+/-3.4% with optimal wavelet and SVM with optimization of the kernel and regularization parameter. The proposed pattern recognition method is promising for classification of MRCPs modulated by variations of force-related parameters.     

Sparse-Representation-Based Classification with Structure-Preserving Dimension Reduction

Sparse-representation-based classification (SRC), which classifies data based on the sparse reconstruction error, has been a new technique in pattern recognition. However, the computation cost for sparse coding is heavy in real applications. In this paper, various dimension reduction methods are studied in the context of SRC to improve classification accuracy as well as reduce computational cost. A feature extraction method, i.e., principal component analysis, and feature selection methods, i.e., Laplacian score and Pearson correlation coefficient, are applied to the data preparation step to preserve the structure of data in the lower-dimensional space. Classification performance of SRC with structure-preserving dimension reduction (SRC–SPDR) is compared to classical classifiers such as k-nearest neighbors and support vector machines. Experimental tests with the UCI and face data sets demonstrate that SRC–SPDR is effective with relatively low computation cost     

View‐centralized multi‐atlas classification for Alzheimer's disease diagnosis

Multi‐atlas based methods have been recently used for classification of Alzheimer's disease (AD) and its prodromal stage, that is, mild cognitive impairment (MCI). Compared with traditional single‐atlas based methods, multiatlas based methods adopt multiple predefined atlases and thus are less biased by a certain atlas. However, most existing multiatlas based methods simply average or concatenate the features from multiple atlases, which may ignore the potentially important diagnosis information related to the anatomical differences among different atlases. In this paper, we propose a novel view (i.e., atlas) centralized multi‐atlas classification method, which can better exploit useful information in multiple feature representations from different atlases. Specifically, all brain images are registered onto multiple atlases individually, to extract feature representations in each atlas space. Then, the proposed view‐centralized multi‐atlas feature selection method is used to select the most discriminative features from each atlas with extra guidance from other atlases. Next, we design a support vector machine (SVM) classifier using the selected features in each atlas space. Finally, we combine multiple SVM classifiers for multiple atlases through a classifier ensemble strategy for making a final decision. We have evaluated our method on 459 subjects [including 97 AD, 117 progressive MCI (p‐MCI), 117 stable MCI (s‐MCI), and 128 normal controls (NC)] from the Alzheimer's Disease Neuroimaging Initiative database, and achieved an accuracy of 92.51% for AD versus NC classification and an accuracy of 78.88% for p‐MCI versus s‐MCI classification. These results demonstrate that the proposed method can significantly outperform the previous multi‐atlas based classification methods. Hum Brain Mapp 36:1847–1865, 2015. © 2015 Wiley Periodicals, Inc .     

Identification of Alzheimer's disease and mild cognitive impairment using multimodal sparse hierarchical extreme learning machine

Different modalities such as structural MRI, FDG‐PET, and CSF have complementary information, which is likely to be very useful for diagnosis of AD and MCI. Therefore, it is possible to develop a more effective and accurate AD/MCI automatic diagnosis method by integrating complementary information of different modalities. In this paper, we propose multi‐modal sparse hierarchical extreme leaning machine (MSH‐ELM). We used volume and mean intensity extracted from 93 regions of interest (ROIs) as features of MRI and FDG‐PET, respectively, and used p‐tau, t‐tau, and as CSF features. In detail, high‐level representation was individually extracted from each of MRI, FDG‐PET, and CSF using a stacked sparse extreme learning machine auto‐encoder (sELM‐AE). Then, another stacked sELM‐AE was devised to acquire a joint hierarchical feature representation by fusing the high‐level representations obtained from each modality. Finally, we classified joint hierarchical feature representation using a kernel‐based extreme learning machine (KELM). The results of MSH‐ELM were compared with those of conventional ELM, single kernel support vector machine (SK‐SVM), multiple kernel support vector machine (MK‐SVM) and stacked auto‐encoder (SAE). Performance was evaluated through 10‐fold cross‐validation. In the classification of AD vs. HC and MCI vs. HC problem, the proposed MSH‐ELM method showed mean balanced accuracies of 96.10% and 86.46%, respectively, which is much better than those of competing methods. In summary, the proposed algorithm exhibits consistently better performance than SK‐SVM, ELM, MK‐SVM and SAE in the two binary classification problems (AD vs. HC and MCI vs. HC).     

A signal theory approach to support vector classification: The sinc kernel

Fourier-based regularisation is considered for the support vector machine classification problem over absolutely integrable loss functions. By invoking the modest assumption that the decision function belongs to a Paley–Wiener space, it is shown that the classification problem can be developed in the context of signal theory. Furthermore, by employing the Paley–Wiener reproducing kernel, namely the sinc function, it is shown that a principled and finite kernel hyper-parameter search space can be discerned, a priori. Subsequent simulations performed on a commonly-available hyperspectral image data set reveal that the approach yields results that surpass state-of-the-art benchmarks.     

SAKM: self-adaptive kernel machine. A kernel-based algorithm for online clustering

This paper presents a new online clustering algorithm called SAKM (Self-Adaptive Kernel Machine) which is developed to learn continuously evolving clusters from non-stationary data. Based on SVM and kernel methods, the SAKM algorithm uses a fast adaptive learning procedure to take into account variations over time. Dedicated to online clustering in a multi-class environment, the algorithm designs an unsupervised neural architecture with self-adaptive abilities. Based on a specific kernel-induced similarity measure, the SAKM learning procedures consist of four main stages: Creation, Adaptation, Fusion and Elimination. In addition to these properties, the SAKM algorithm is attractive to be computationally efficient in online learning of real-drifting targets. After a theoretical study of the error convergence bound of the SAKM local learning, a comparison with NORMA and ALMA algorithms is made. In the end, some experiments conducted on simulation data, UCI benchmarks and real data are given to illustrate the capacities of the SAKM algorithm for online clustering in non-stationary and multi-class environment.     

Robust Bayesian Classification with Incomplete Data

In this paper, we address the Bayesian classification with incomplete data. The common approach in the literature is to simply ignore the samples with missing values or impute missing values before classification. However, these methods are not effective when a large portion of the data have missing values and the acquisition of samples is expensive. Motivated by these limitations, the expectation maximization algorithm for learning a multivariate Gaussian mixture model and a multiple kernel density estimator based on the propensity scores are proposed to avoid listwise deletion (LD) or mean imputation (MI) for solving classification tasks with incomplete data. We illustrate the effectiveness of our proposed algorithms on some artificial and benchmark UCI data sets by comparing with LD and MI methods. We also apply these algorithms to solve the practical classification tasks on the lithology identification of hydrothermal minerals and license plate character recognition. The experimental results demonstrate their good performance with high classification accuracies.     

Robust support vector machine-trained fuzzy system

Because the SVM (support vector machine) classifies data with the widest symmetric margin to decrease the probability of the test error, modern fuzzy systems use SVM to tune the parameters of fuzzy if–then rules. But, solving the SVM model is time-consuming. To overcome this disadvantage, we propose a rapid method to solve the robust SVM model and use it to tune the parameters of fuzzy if–then rules. The robust SVM is an extension of SVM for interval-valued data classification.We compare our proposed method with SVM, robust SVM, ISVM-FC (incremental support vector machine-trained fuzzy classifier), BSVM-FC (batch support vector machine-trained fuzzy classifier), SOTFN-SV (a self-organizing TS-type fuzzy network with support vector learning) and SCLSE (a TS-type fuzzy system with subtractive clustering for antecedent parameter tuning and LSE for consequent parameter tuning) by using some real datasets. According to experimental results, the use of proposed approach leads to very low training and testing time with good misclassification rate.     

基于遗传算法与支持向量机的基因微阵列分析

背景：微阵列数据的特点是样本含量小，而变量数(基因)多达上万个。此时，传统的统计方法往往因为高维而失效了。遗传算法和支持向量机是近年来发展迅速的机器学习算法，具有很好的分类效果与降维优势。 目的：提出将遗传算法与支持向量机结合起来对样本进行分类，并与直接采用支持向量机、筛选差异表达基因后采用支持向量机的结果进行比较。 方法：采用Bioconductor提供的数据集golub,它是白血病微阵列芯片实验所得的基因表达数据集，对全部基因采用支持向量机进行分类。采用SAM软件对芯片数据的显著性分析确定不同的差异表达基因并估计错误发现率FDR，以筛选出的76个差异表达基因作为特征基因子集，再采用支持向量机进行分类。将筛选出的76个差异表达基因作为初始的特征基因集合，采用遗传算法－支持向量机再次进行特征基因选择，提高分类准确度，并与全部基因直接采用支持向量机、筛选差异表达基因后采用支持向量机的结果进行比较。同时也对特征基因在代谢通路上的分布和功能作了一定的研究。 结果与结论：通过遗传算法降维可以提高支持向量机的分类准确率，特别是剔除了数据中的大量无关基因和噪声，使得经过特征选择后分类准确率提高。结果显示遗传算法与支持向量机结合方法对分类更加有效。此外，通路分析结果显示特征基因的主要功能体现在信号传导和氨基酸代谢上。     

Semi-supervised learning for tree-structured ensembles of RBF networks with Co-Training

Supervised learning requires a large amount of labeled data, but the data labeling process can be expensive and time consuming, as it requires the efforts of human experts. Co-Training is a semi-supervised learning method that can reduce the amount of required labeled data through exploiting the available unlabeled data to improve the classification accuracy. It is assumed that the patterns are represented by two or more redundantly sufficient feature sets (views) and these views are independent given the class. On the other hand, most of the real-world pattern recognition tasks involve a large number of categories which may make the task difficult. The tree-structured approach is an output space decomposition method where a complex multi-class problem is decomposed into a set of binary sub-problems. In this paper, we propose two learning architectures to combine the merits of the tree-structured approach and Co-Training. We show that our architectures are especially useful for classification tasks that involve a large number of classes and a small amount of labeled data where the single-view tree-structured approach does not perform well alone but when combined with Co-Training, it can exploit effectively the independent views and the unlabeled data to improve the recognition accuracy.     

Enhanced active segment selection for single-trial EEG classification

In this study, an electroencephalogram (EEG) analysis system is proposed for single-trial classification of both motor imagery (MI) and finger-lifting EEG data. Applying event-related brain potential (ERP) data acquired from the sensorimotor cortices, the system mainly consists of three procedures; enhanced active segment selection, feature extraction, and classification. In addition to the original use of continuous wavelet transform (CWT) and Student 2-sample t statistics, the two-dimensional (2D) anisotropic Gaussian filter further refines the selection of active segments. The multiresolution fractal features are then extracted from wavelet data by using proposed modified fractal dimension. Finally, the support vector machine (SVM) is used for classification. Compared to original active segment selection, with several popular features and classifier on both the MI and finger-lifting data from 2 data sets, the results indicate that the proposed method is promising in EEG classification.     

The context-tree kernel for strings.

We propose a new kernel for strings which borrows ideas and techniques from information theory and data compression. This kernel can be used in combination with any kernel method, in particular Support Vector Machines for string classification, with notable applications in proteomics. By using a Bayesian averaging framework with conjugate priors on a class of Markovian models known as probabilistic suffix trees or context-trees, we compute the value of this kernel in linear time and space while only using the information contained in the spectrum of the considered strings. This is ensured through an adaptation of a compression method known as the context-tree weighting algorithm. Encouraging classification results are reported on a standard protein homology detection experiment, showing that the context-tree kernel performs well with respect to other state-of-the-art methods while using no biological prior knowledge.     

Predictive Models Based on Support Vector Machines: Whole‐Brain versus Regional Analysis of Structural MRI in the Alzheimer's Disease

Decision‐making systems trained on structural magnetic resonance imaging data of subjects affected by the Alzheimer's disease (AD) and healthy controls (CTRL) are becoming widespread prognostic tools for subjects with mild cognitive impairment (MCI). This study compares the performances of three classification methods based on support vector machines (SVMs), using as initial sets of brain voxels (ie, features): (1) the segmented grey matter (GM); (2) regions of interest (ROIs) by voxel‐wise t‐test filtering; (3) parceled ROIs, according to prior knowledge. The recursive feature elimination (RFE) is applied in all cases to investigate whether feature reduction improves the classification accuracy. We analyzed more than 600 AD Neuroimaging Initiative (ADNI) subjects, training the SVMs on the AD/CTRL dataset, and evaluating them on a trial MCI dataset. The classification performance, evaluated as the area under the receiver operating characteristic curve (AUC), reaches AUC = (88.9 ± .5)% in 20‐fold cross‐validation on the AD/CTRL dataset, when the GM is classified as a whole. The highest discrimination accuracy between MCI converters and nonconverters is achieved when the SVM‐RFE is applied to the whole GM: with AUC reaching (70.7 ± .9)%, it outperforms both ROI‐based approaches in predicting the AD conversion.     

Comparison of Feature Selection Techniques in Machine Learning for Anatomical Brain MRI in Dementia

We present a comparative split-half resampling analysis of various data driven feature selection and classification methods for the whole brain voxel-based classification analysis of anatomical magnetic resonance images. We compared support vector machines (SVMs), with or without filter based feature selection, several embedded feature selection methods and stability selection. While comparisons of the accuracy of various classification methods have been reported previously, the variability of the out-of-training sample classification accuracy and the set of selected features due to independent training and test sets have not been previously addressed in a brain imaging context. We studied two classification problems: 1) Alzheimer’s disease (AD) vs. normal control (NC) and 2) mild cognitive impairment (MCI) vs. NC classification. In AD vs. NC classification, the variability in the test accuracy due to the subject sample did not vary between different methods and exceeded the variability due to different classifiers. In MCI vs. NC classification, particularly with a large training set, embedded feature selection methods outperformed SVM-based ones with the difference in the test accuracy exceeding the test accuracy variability due to the subject sample. The filter and embedded methods produced divergent feature patterns for MCI vs. NC classification that suggests the utility of the embedded feature selection for this problem when linked with the good generalization performance. The stability of the feature sets was strongly correlated with the number of features selected, weakly correlated with the stability of classification accuracy, and uncorrelated with the average classification accuracy.