Using Siamese capsule networks for remote sensing scene classification

ABSTRACT

The convolutional neural network (CNN) is widely used for image classification because of its powerful feature extraction capability. The key challenge of CNN in remote sensing (RS) scene classification is that the size of data set is small and images in each category vary greatly in position and angle, while the spatial information will be lost in the pooling layers of CNN. Consequently, how to extract accurate and effective features is very important. To this end, we present a Siamese capsule network to address these issues. Firstly, we introduce capsules to extract the spatial information of the features so as to learn equivariant representations. Secondly, to improve the classification accuracy of the model on small data sets, the proposed model utilizes the structure of the Siamese network as embedded verification. Finally, the features learned through Capsule networks are regularized by a metric learning term to improve the robustness of our model. The effectiveness of the model on three benchmark RS data sets is verified by different experiments. Experimental results demonstrate that the comprehensive performance of the proposed method surpasses other existing methods.     

Parallel multi-stage features fusion of deep convolutional neural networks for aerial scene classification

Aerial scene classification is a challenging task in the remote sensing image processing field. Owing to some similar scene, there are only differences in density. To challenge this problem, this paper proposes a novel parallel multi-stage (PMS) architecture formed by a low, middle, and high deep convolutional neural network (DCNN) sub-model. PMS model automatically learns representative and discriminative hierarchical features, which include three 512 dimension vectors, respectively, and the final representative feature created by linear connection. PMS model describes a robust feature of aerial image through three stages feature. Unlike previous methods, we only use transfer learning and deep learning methods to obtain more discriminative features from scene images while improving performance. Experimental results demonstrate that the proposed PMS model has a more superior performance than the state-of-the-art methods, obtaining average classification accuracies of 98.81% and 95.56%, respectively, on UC Merced (UCM) and aerial image dataset (AID) benchmark datasets.     

Selective convolutional neural networks and cascade classifiers for remote sensing image classification

Training convolutional neural network (CNN) architecture fully, using pretrained CNNs as feature extractors, and fine-tuning pretrained CNNs on target datasets are three popular strategies used in state-of-the-art methods for remote sensing image classification. The full training strategy requires large-scale training dataset, whereas the fine-tuning strategy requires a pretrained model to resume network learning. In this study, we propose a new strategy based on selective CNNs and cascade classifiers to improve the classification accuracy of remote sensing images relative to single CNN. First, we conduct a comparative study of existing pretrained CNNs in terms of data augmentation and the use of fully connected layers. Second, selective CNNs, which based on class separability criterion, are presented to obtain an optimal combination from multiple pretrained models. Finally, classification accuracy is improved by introducing two-stage cascade linear classifiers, the prediction probability of which in the first stage is used as input for the second stage. Experiments on three public remote sensing datasets demonstrate the effectiveness of the proposed method in comparison with state-of-the-art methods.     

A deep learning framework for hyperspectral image classification using spatial pyramid pooling

In this letter, a new deep learning framework for spectral–spatial classification of hyperspectral images is presented. The proposed framework serves as an engine for merging the spatial and spectral features via suitable deep learning architecture: stacked autoencoders (SAEs) and deep convolutional neural networks (DCNNs) followed by a logistic regression (LR) classifier. In this framework, SAEs is aimed to get useful high-level features for the one-dimensional features which is suitable for the dimension reduction of spectral features, while DCNNs can learn rich features from the training data automatically and has achieved state-of-the-art performance in many image classification databases. Though the DCNNs has shown robustness to distortion, it only extracts features of the same scale, and hence is insufficient to tolerate large-scale variance of object. As a result, spatial pyramid pooling (SPP) is introduced into hyperspectral image classification for the first time by pooling the spatial feature maps of the top convolutional layers into a fixed-length feature. Experimental results with widely used hyperspectral data indicate that classifiers built in this deep learning-based framework provide competitive performance.     

Micro-Net: A unified model for segmentation of various objects in microscopy images

Object segmentation and structure localization are important steps in automated image analysis pipelines for microscopy images. We present a convolution neural network (CNN) based deep learning architecture for segmentation of objects in microscopy images. The proposed network can be used to segment cells, nuclei and glands in fluorescence microscopy and histology images after slight tuning of input parameters. The network trains at multiple resolutions of the input image, connects the intermediate layers for better localization and context and generates the output using multi-resolution deconvolution filters. The extra convolutional layers which bypass the max-pooling operation allow the network to train for variable input intensities and object size and make it robust to noisy data. We compare our results on publicly available data sets and show that the proposed network outperforms recent deep learning algorithms.     

A multi-depth convolutional neural network for SAR image classification

The convolutional neural network has been widely used in synthetic aperture radar (SAR) image classification, for it can learn discriminative features from massive amounts of data. However, it is short of distinctive learning mechanisms for different regions in SAR images. In this letter, a novel architecture called multi-depth convolutional neural network (Multi-depth CNN) is proposed which can select different levels of features for classification. Differing from classical convolutional neural network, Multi-depth CNN adopts a piecewise back-propagation method to optimize the network. Meanwhile, compared with classical convolutional neural network, the proposed network can reduce the training time effectively. Experimental results on two datasets demonstrate that the proposed network can achieve better classification accuracy compared with some state-of-art algorithms.     

Deep learning classification of cervical dysplasia using depth-resolved angular light scattering profiles

We present a machine learning method for detecting and staging cervical dysplastic tissue using light scattering data based on a convolutional neural network (CNN) architecture. Depth-resolved angular scattering measurements from two clinical trials were used to generate independent training and validation sets as input of our model. We report 90.3% sensitivity, 85.7% specificity, and 87.5% accuracy in classifying cervical dysplasia, showing the uniformity of classification of a/LCI scans across different instruments. Further, our deep learning approach significantly improved processing speeds over the traditional Mie theory inverse light scattering analysis (ILSA) method, with a hundredfold reduction in processing time, offering a promising approach for a/LCI in the clinic for assessing cervical dysplasia.     

ResGANet: Residual group attention network for medical image classification and segmentation

In recent years, deep learning technology has shown superior performance in different fields of medical image analysis. Some deep learning architectures have been proposed and used for computational pathology classification, segmentation, and detection tasks. Due to their simple, modular structure, most downstream applications still use ResNet and its variants as the backbone network. This paper proposes a modular group attention block that can capture feature dependencies in medical images in two independent dimensions: channel and space. By stacking these group attention blocks in ResNet-style, we obtain a new ResNet variant called ResGANet. The stacked ResGANet architecture has 1.51–3.47 times fewer parameters than the original ResNet and can be directly used for downstream medical image segmentation tasks. Many experiments show that the proposed ResGANet is superior to state-of-the-art backbone models in medical image classification tasks. Applying it to different segmentation networks can improve the baseline model in medical image segmentation tasks without changing the network architecture. We hope that this work provides a promising method for enhancing the feature representation of convolutional neural networks (CNNs) in the future.     

Recognition of targets in SAR images using joint classification of deep features fused by multi-canonical correlation analysis

This letter proposes a synthetic aperture radar (SAR) target recognition method via joint classification of deep features fused by multi-canonical correlation analysis (MCCA). A convolutional neural network (CNN) is designed for feature learning from original SAR images. For the multiple feature maps from different convolution layers, they are fused based on the MCCA to maintain the relevance while eliminating the redundancy. Afterwards, the joint sparse representation (JSR) is employed to jointly represent the fused deep feature vectors from different convolution layers under the constraint of their inner correlations. Based on the reconstruction errors from JSR, the target label can be classified. The proposed method can make full use of the multi-level deep features by using the correlations among the same layer and between different layers. Experiments are investigated on the Moving and Stationary Target Acquisition and Recognition (MSTAR) data set and the results confirm the performance of the proposed method.     

Hyperspectral image classification based on convolutional neural network and random forest

ABSTRACT

Deep learning-based methods, especially deep convolutional neural network (CNN), have proven their powerfulness in hyperspectral image (HSI) classification. On the other hand, ensemble learning is a useful method for classification task. In this letter, in order to further improve the classification accuracy, the combination of CNN and random forest (RF) is proposed for HSI classification. The well-designed CNN is used as individual classifier to extract the discriminant features of HSI and RF randomly selects the extracted features and training samples to formulate a multiple classifier system. Furthermore, the learned weights of CNN are adopted to initialize other individual CNN. Experimental results with two hyperspectral data sets indicate that the proposed method provides competitive classification results compared with state-of-the-art methods.     

Hyperspectral images classification with convolutional neural network and textural feature using limited training samples

In this letter, a new deep learning framework, which integrates textural features of gray level co-occurrence matrix (GLCM) into convolutional neural networks (CNNs) is proposed for hyperspectral images (HSIs) classification using limited number of labeled samples. The proposed method can be implemented in three steps. Firstly, the GLCM textural features are extracted from the first principal component after the principal components analysis (PCA) transformation. Secondly, a CNN is built to extract the deep spectral features from the original HSIs, and the features are concatenated with the textural features obtained in the first step in a concat layer of CNN. Finally, softmax is employed to generate classification maps at the end of the framework. In this way, the CNN focuses on the learning of spectral features only, and the generated textural features are used directly as one set of features before softmax. These lead to the reduction of the requirements for the size of training samples and the improvement of computing efficiency. The experimental results are presented for three HSIs and compared with several advanced deep learning and spectral-spatial classification techniques. The competitive classification accuracy can be obtained, especially when only a limited number of training samples are available.     

Semi-supervised deep learning for hyperspectral image classification

Recently, a series of deep learning methods based on the convolutional neural networks (CNNs) have been introduced for classification of hyperspectral images (HSIs). However, in order to obtain the optimal parameters, a large number of training samples are required in the CNNs to avoid the overfitting problem. In this paper, a novel method is proposed to extend the training set for deep learning based hyperspectral image classification. First, given a small-sample-size training set, the principal component analysis based edge-preserving features (PCA-EPFs) and extended morphological attribute profiles (EMAPs) are used for HSI classification so as to generate classification probability maps. Second, a large number of pseudo training samples are obtained by the designed decision function which depends on the classification probabilities. Finally, a deep feature fusion network (DFFN) is applied to classify HSI with the training set consists of the original small-sample-size training set and the added pseudo training samples. Experiments performed on several hyperspectral data sets demonstrate the state-of-the-art performance of the proposed method in terms of classification accuracies.     

Identification of mine water inrush using laser-induced fluorescence spectroscopy combined with one-dimensional convolutional neural network

The application of laser-induced fluorescence (LIF) combined with machine learning methods can make up for the shortcomings of traditional hydrochemical methods in the accurate and rapid identification of mine water inrush in coal mines. However, almost all of these methods require preprocessing such as principal component analysis (PCA) or drawing the spectral map as an essential step. Here, we provide our solution for the classification of mine water inrush, in which a one-dimensional convolutional neural network (1D CNN) is trained to automatically identify mine water inrush according to the LIF spectroscopy without the need for preprocessing. First, the architecture and parameters of the model were optimized and the 1D CNN model containing two convolutional blocks was determined to be the best model for the identification of mine water inrush. Then, we evaluated the performance of the 1D CNN model using the LIF spectral dataset of mine water inrush containing 540 training samples and 135 test samples, and we found that all 675 samples could be accurately identified. Finally, superior classification performance was demonstrated by comparing with a traditional machine learning algorithm (genetic algorithm-support vector machine) and a deep learning algorithm (two-dimensional convolutional neural network). The results show that LIF spectroscopy combined with 1D CNN can be used for the fast and accurate identification of mine water inrush without the need for complex pretreatments.

LIF spectroscopy combined with 1D CNN can identify mine water inrush quickly and accurately without complicated pretreatment.     

Convolutional neural networks for classification of Alzheimer's disease: Overview and reproducible evaluation

Numerous machine learning (ML) approaches have been proposed for automatic classification of Alzheimer's disease (AD) from brain imaging data. In particular, over 30 papers have proposed to use convolutional neural networks (CNN) for AD classification from anatomical MRI. However, the classification performance is difficult to compare across studies due to variations in components such as participant selection, image preprocessing or validation procedure. Moreover, these studies are hardly reproducible because their frameworks are not publicly accessible and because implementation details are lacking. Lastly, some of these papers may report a biased performance due to inadequate or unclear validation or model selection procedures. In the present work, we aim to address these limitations through three main contributions. First, we performed a systematic literature review. We identified four main types of approaches: i) 2D slice-level, ii) 3D patch-level, iii) ROI-based and iv) 3D subject-level CNN. Moreover, we found that more than half of the surveyed papers may have suffered from data leakage and thus reported biased performance. Our second contribution is the extension of our open-source framework for classification of AD using CNN and T1-weighted MRI. The framework comprises previously developed tools to automatically convert ADNI, AIBL and OASIS data into the BIDS standard, and a modular set of image preprocessing procedures, classification architectures and evaluation procedures dedicated to deep learning. Finally, we used this framework to rigorously compare different CNN architectures. The data was split into training/validation/test sets at the very beginning and only the training/validation sets were used for model selection. To avoid any overfitting, the test sets were left untouched until the end of the peer-review process. Overall, the different 3D approaches (3D-subject, 3D-ROI, 3D-patch) achieved similar performances while that of the 2D slice approach was lower. Of note, the different CNN approaches did not perform better than a SVM with voxel-based features. The different approaches generalized well to similar populations but not to datasets with different inclusion criteria or demographical characteristics. All the code of the framework and the experiments is publicly available: general-purpose tools have been integrated into the Clinica software (www.clinica.run) and the paper-specific code is available at: https://github.com/aramis-lab/AD-DL.     

Vehicle detection in remote sensing images using denoizing-based convolutional neural networks

Vehicle detection in remote sensing images is a tough task and of great significance due to the fast increasing number of vehicles occurring in big cities. Recently, convolutional neural network (CNN)-based methods have achieved excellent performance in classification task due to their powerful abilities in high-level feature extraction. However, overfitting is a serious problem in CNN when applying complicated fully-connected layers, especially when the quantity of training samples is limited. In order to tackle this problem, a denoizing-based CNN called DCNN is proposed in this letter. More specially, a CNN with one fully-connected layer is pre-trained first for feature extraction. After that, features of this fully-connected layer are corrupted and used to pre-train a stacked denoizing autoencoder (SDAE) in an unsupervised way. Then, the pre-trained SDAE is added into the CNN as the fully-connected layer. After fine-tuning, DCNN can make the extracted features more robust and the detecting rate higher. With the help of our proposed locating method, vehicles can be detected effectively even when they are parked in a residential area. Comparative experiments demonstrate that our method has achieved state-of-the-art performance.     

Classification of crystal structures using electron diffraction patterns with a deep convolutional neural network

Investigations have been made to explore the applicability of an off-the-shelf deep convolutional neural network (DCNN) architecture, residual neural network (ResNet), to the classification of the crystal structure of materials using electron diffraction patterns without prior knowledge of the material systems under consideration. The dataset required for training and validating the ResNet architectures was obtained by the computer simulation of the selected area electron diffraction (SAD) in transmission electron microscopy. Acceleration voltages, zone axes, and camera lengths were used as variables and crystal information format (CIF) files obtained from open crystal data repositories were used as inputs. The cubic crystal system was chosen as a model system and five space groups of 213, 221, 225, 227, and 229 in the cubic system were selected for the test and validation, based on the distinguishability of the SAD patterns. The simulated diffraction patterns were regrouped and labeled from the viewpoint of computer vision, i.e., the way how the neural network recognizes the two-dimensional representation of three-dimensional lattice structure of crystals, for improved training and classification efficiency. Comparison of the various ResNet architectures with varying number of layers demonstrated that the ResNet101 architecture could classify the space groups with the validation accuracy of 92.607%.

The off-the-shelf deep convolutional neural network architecture, ResNet, could classify the space group of materials with cubic crystal structures with the prediction accuracy of 92.607%, using the selected area electron diffraction patterns.     

Attention-Enriched Deep Learning Model for Breast Tumor Segmentation in Ultrasound Images

Incorporating human domain knowledge for breast tumor diagnosis is challenging because shape, boundary, curvature, intensity or other common medical priors vary significantly across patients and cannot be employed. This work proposes a new approach to integrating visual saliency into a deep learning model for breast tumor segmentation in ultrasound images. Visual saliency refers to image maps containing regions that are more likely to attract radiologists’ visual attention. The proposed approach introduces attention blocks into a U-Net architecture and learns feature representations that prioritize spatial regions with high saliency levels. The validation results indicate increased accuracy for tumor segmentation relative to models without salient attention layers. The approach achieved a Dice similarity coefficient (DSC) of 90.5% on a data set of 510 images. The salient attention model has the potential to enhance accuracy and robustness in processing medical images of other organs, by providing a means to incorporate task-specific knowledge into deep learning architectures.     

人工智能辅助床旁超声诊断肾创伤的基础研究

目的探讨基于卷积神经网络（CNN）构建的人工智能辅助诊断模型对肾钝性创伤超声诊断的应用价值。 方法建立不同程度动物肾创伤模型,通过床旁超声仪采集正常肾及创伤肾超声图片,分成训练集及测试集,根据造模位置和超声造影结果,手动勾画出肾轮廓,采用3折交叉验证进行分类训练及测试。绘制受试者工作特征（ROC）曲线,计算人工智能辅助诊断模型的敏感度、特异度、准确性和曲线下面积（AUC）。 结果采集正常肾图片共1737张,各级别创伤肾图片共2125张,经过对测试集的验证,该模型可自动对肾创伤有无进行分类,对肾创伤诊断的平均敏感度为73%、平均特异度为85%、平均准确性为79%、AUC为0.80,诊断价值较高。 结论基于CNN构建的深度学习模型辅助床旁超声仪在诊断肾创伤有无分类中取得了较满意的结果。     

Breast Cancer Classification in Automated Breast Ultrasound Using Multiview Convolutional Neural Network with Transfer Learning

To assist radiologists in breast cancer classification in automated breast ultrasound (ABUS) imaging, we propose a computer-aided diagnosis based on a convolutional neural network (CNN) that classifies breast lesions as benign and malignant. The proposed CNN adopts a modified Inception-v3 architecture to provide efficient feature extraction in ABUS imaging. Because the ABUS images can be visualized in transverse and coronal views, the proposed CNN provides an efficient way to extract multiview features from both views. The proposed CNN was trained and evaluated on 316 breast lesions (135 malignant and 181 benign). An observer performance test was conducted to compare five human reviewers' diagnostic performance before and after referring to the predicting outcomes of the proposed CNN. Our method achieved an area under the curve (AUC) value of 0.9468 with five-folder cross-validation, for which the sensitivity and specificity were 0.886 and 0.876, respectively. Compared with conventional machine learning-based feature extraction schemes, particularly principal component analysis (PCA) and histogram of oriented gradients (HOG), our method achieved a significant improvement in classification performance. The proposed CNN achieved a >10% increased AUC value compared with PCA and HOG. During the observer performance test, the diagnostic results of all human reviewers had increased AUC values and sensitivities after referring to the classification results of the proposed CNN, and four of the five human reviewers’ AUCs were significantly improved. The proposed CNN employing a multiview strategy showed promise for the diagnosis of breast cancer, and could be used as a second reviewer for increasing diagnostic reliability.