Remote sensing image scene classification based on generative adversarial networks

Scene classification of remote sensing images plays an important role in many remote sensing image applications. Training a good classifier needs a large number of training samples. The labeled samples are often scarce and difficult to obtain, and annotating a large number of samples is time-consuming. We propose a novel remote sensing image scene classification framework based on generative adversarial networks (GAN) in this paper. GAN can improve the generalization ability of machine learning network model. However, generating large-size images, especially high-resolution remote sensing images is difficult. To address this issue, the scaled exponential linear units (SELU) are applied into the GAN to generate high quality remote sensing images. Experiments carried out on two datasets show that our approach can obtain the state-of-the-art results compared with the classification results of the classic deep convolutional neural networks, especially when the number of training samples is small.     

Application of a parallel spectral–spatial convolution neural network in object-oriented remote sensing land use classification

Due to the abundance of spatial information and relative lack of spectral information in high spatial resolution remote sensing images, a land use classification method for high-resolution remote sensing images is proposed based on a parallel spectral-spatial convolutional neural network (CNN) and object-oriented remote sensing technology. The contour of a remote sensing object is taken as the boundary and the set of pixels that comprises the object are extracted to form the input data set for the deep neural network. The proposed network considers both the features of the object and the pixels which forms the object. The spatial and spectral features of remote sensing image objects are extracted independently in the parallel network using panchromatic and multispectral remote sensing techniques. Then, through a fully connected layer, both spectral and spatial information are integrated to produce remote sensing object class coding. The experimental results demonstrate that the parallel spectral-spatial CNN, which combines spatial and spectral features, achieves better classification performance than the individual CNN. Therefore, the proposed method provides a novel approach to land use classification based on high spatial resolution remote sensing images.     

Fine-grained ship classification based on deep residual learning for high-resolution SAR images

ABSTRACT

As the resolution of Synthetic Aperture Radar (SAR) images increases, the fine-grained classification of ships has become a focus of the SAR field. In this paper, a ship classification framework based on deep residual network for high-resolution SAR images is proposed. In general, networks with more layers have higher classification accuracy. However, the training accuracy degradation and the limited dataset are major problems in the training process. To build deeper networks, residual modules are constructed and batch normalization is applied to keep the activation function output. Different fine tuning strategies are used to select the best training scheme. To take advantage of the proposed framework, a dataset including 835 ship slices is augmented by different multiples and then used to validate our method and other Convolutional Neural Network (CNN) models. The experimental results show that, the proposed framework can achieve a 99% overall accuracy on the augmented dataset under the optimal fine-tuning strategy, 3% higher than that in other models, which demonstrates the effectiveness of our proposed approach.     

Vehicle detection in remote sensing images using denoizing-based convolutional neural networks

Vehicle detection in remote sensing images is a tough task and of great significance due to the fast increasing number of vehicles occurring in big cities. Recently, convolutional neural network (CNN)-based methods have achieved excellent performance in classification task due to their powerful abilities in high-level feature extraction. However, overfitting is a serious problem in CNN when applying complicated fully-connected layers, especially when the quantity of training samples is limited. In order to tackle this problem, a denoizing-based CNN called DCNN is proposed in this letter. More specially, a CNN with one fully-connected layer is pre-trained first for feature extraction. After that, features of this fully-connected layer are corrupted and used to pre-train a stacked denoizing autoencoder (SDAE) in an unsupervised way. Then, the pre-trained SDAE is added into the CNN as the fully-connected layer. After fine-tuning, DCNN can make the extracted features more robust and the detecting rate higher. With the help of our proposed locating method, vehicles can be detected effectively even when they are parked in a residential area. Comparative experiments demonstrate that our method has achieved state-of-the-art performance.     

A Y-Net deep learning method for road segmentation using high-resolution visible remote sensing images

Road segmentation from high-resolution visible remote sensing images provides an effective way for automatic road network forming. Recently, deep learning methods based on convolutional neural networks (CNNs) are widely applied in road segmentation. However, it is a challenge for most CNN-based methods to achieve high segmentation accuracy when processing high-resolution visible remote sensing images with rich details. To handle this problem, we propose a road segmentation method based on a Y-shaped convolutional network (indicated as Y-Net). Y-Net contains a two-arm feature extraction module and a fusion module. The feature extraction module includes a deep downsampling-to-upsampling sub-network for semantic features and a convolutional sub-network without downsampling for detail features. The fusion module combines all features for road segmentation. Benefiting from this scheme, the Y-Net can well segment multi-scale roads (both wide and narrow roads) from high-resolution images. The testing and comparative experiments on a public dataset and a private dataset show that Y-Net has higher segmentation accuracy than four other state-of-art methods, FCN (Fully Convolutional Network), U-Net, SegNet, and FC-DenseNet (Fully Convolutional DenseNet). Especially, Y-Net accurately segments contours of narrow roads, which are missed by the comparative methods.     

Selective convolutional neural networks and cascade classifiers for remote sensing image classification

Training convolutional neural network (CNN) architecture fully, using pretrained CNNs as feature extractors, and fine-tuning pretrained CNNs on target datasets are three popular strategies used in state-of-the-art methods for remote sensing image classification. The full training strategy requires large-scale training dataset, whereas the fine-tuning strategy requires a pretrained model to resume network learning. In this study, we propose a new strategy based on selective CNNs and cascade classifiers to improve the classification accuracy of remote sensing images relative to single CNN. First, we conduct a comparative study of existing pretrained CNNs in terms of data augmentation and the use of fully connected layers. Second, selective CNNs, which based on class separability criterion, are presented to obtain an optimal combination from multiple pretrained models. Finally, classification accuracy is improved by introducing two-stage cascade linear classifiers, the prediction probability of which in the first stage is used as input for the second stage. Experiments on three public remote sensing datasets demonstrate the effectiveness of the proposed method in comparison with state-of-the-art methods.     

Using Siamese capsule networks for remote sensing scene classification

ABSTRACT

The convolutional neural network (CNN) is widely used for image classification because of its powerful feature extraction capability. The key challenge of CNN in remote sensing (RS) scene classification is that the size of data set is small and images in each category vary greatly in position and angle, while the spatial information will be lost in the pooling layers of CNN. Consequently, how to extract accurate and effective features is very important. To this end, we present a Siamese capsule network to address these issues. Firstly, we introduce capsules to extract the spatial information of the features so as to learn equivariant representations. Secondly, to improve the classification accuracy of the model on small data sets, the proposed model utilizes the structure of the Siamese network as embedded verification. Finally, the features learned through Capsule networks are regularized by a metric learning term to improve the robustness of our model. The effectiveness of the model on three benchmark RS data sets is verified by different experiments. Experimental results demonstrate that the comprehensive performance of the proposed method surpasses other existing methods.     

A segmented particle swarm optimization convolutional neural network for land cover and land use classification of remote sensing images

ABSTRACT

The development of automatic classification methods in neural networks is an important topic in the field of land cover and land use (LULC) classification of remote sensing images. Here, we proposed a new segmented particle swarm convolutional neural network model (SPSO-CNN) by combining the subsection particle swarm algorithm with a convolutional neural network. The SPSO-CNN was applied to experiment of LULC classification of GF-1 high resolution remote sensing image. The results showed that SPSO-CNN achieved high precision, recall, F1 score and total precision in the LULC classification of remote sensing image with high spatial resolution, demonstrating the advantage and potential of applying SPSO-CNN to the LULC classification of remote sensing images.     

Large scale deep learning for computer aided detection of mammographic lesions

Recent advances in machine learning yielded new techniques to train deep neural networks, which resulted in highly successful applications in many pattern recognition tasks such as object detection and speech recognition. In this paper we provide a head-to-head comparison between a state-of-the art in mammography CAD system, relying on a manually designed feature set and a Convolutional Neural Network (CNN), aiming for a system that can ultimately read mammograms independently. Both systems are trained on a large data set of around 45,000 images and results show the CNN outperforms the traditional CAD system at low sensitivity and performs comparable at high sensitivity. We subsequently investigate to what extent features such as location and patient information and commonly used manual features can still complement the network and see improvements at high specificity over the CNN especially with location and context features, which contain information not available to the CNN. Additionally, a reader study was performed, where the network was compared to certified screening radiologists on a patch level and we found no significant difference between the network and the readers.     

Heartbeat monitoring with an mm-wave radar based on deep learning: a novel approach for training and classifying heterogeneous signals

ABSTRACT

Millimetre wave radar is an emerging technology that can monitor vital signs without contact. This unique feature is very suitable for some particular situations, such as burn patient monitoring. Currently, electrocardiogram (ECG) is still the most common approach for monitoring heart disease. Deep learning algorithms have already been applied to classifying ECG recordings and have achieved good diagnostic results. However, it is very rare to see deep learning-based heartbeat classification using radar signals. The reason is a lack of radar-based heart disease datasets, which are the most important part of training a Convolutional Neural Network (CNN). Specifically, the ECG recordings and radar signals are heterogeneous; thus, the ECG dataset cannot train the CNN for directly classifying the radar signals. In this paper, we propose a novel signal processing algorithm called the Common Features Extraction Method (CFEM) to extract the common features of ECG recordings and radar signals to train a CNN for radar heartbeat signal classification. By using CFEM, the ECG dataset is transferred to the radar field, which means that the core issue for training the CNN using radar signals has been solved. Practical experiments show that the CFEM-based CNN can classify heartbeat radar signals accurately.     

Focused Attention in Transformers for interpretable classification of retinal images

Vision Transformers have recently emerged as a competitive architecture in image classification. The tremendous popularity of this model and its variants comes from its high performance and its ability to produce interpretable predictions. However, both of these characteristics remain to be assessed in depth on retinal images. This study proposes a thorough performance evaluation of several Transformers compared to traditional Convolutional Neural Network (CNN) models for retinal disease classification. Special attention is given to multi-modality imaging (fundus and OCT) and generalization to external data. In addition, we propose a novel mechanism to generate interpretable predictions via attribution maps. Existing attribution methods from Transformer models have the disadvantage of producing low-resolution heatmaps. Our contribution, called Focused Attention, uses iterative conditional patch resampling to tackle this issue. By means of a survey involving four retinal specialists, we validated both the superior interpretability of Vision Transformers compared to the attribution maps produced from CNNs and the relevance of Focused Attention as a lesion detector.     

A multi-model ensemble method based on convolutional neural networks for aircraft detection in large remote sensing images

Aircraft detection is one of hot issues of remote sensing (RS) image interpretation. Different from existing methods which aim at detecting objects in regions of interest (ROI) of airports, in this work, we propose a novel and effective aircraft detection framework based on convolutional neural networks (CNN) to detect multi-scale targets in extremely large and complicated scenes. In particular, we design a constrained EdgeBoxes (CEdgeBoxes) approach to generate a modest number of target candidates quickly and precisely. Then, in order to overcome the drawbacks of using handcrafted features and traditional classifiers, a modified GoogLeNet combined with Fast Region-based CNN (R-CNN) is designed to extract useful features from RS images and then detect various kinds of multi-scale aircrafts. Most importantly, we propose a new multi-model ensemble method to decrease the false alarm rate (FAR) caused by the imbalanced distribution of complicated targets and backgrounds. Thorough experiments, which are conducted on the dataset acquired by QuickBird with a resolution of 0.6 m, demonstrate our method can effectively detect aircrafts in large scenes with low FARs.     

Parallel multi-stage features fusion of deep convolutional neural networks for aerial scene classification

Aerial scene classification is a challenging task in the remote sensing image processing field. Owing to some similar scene, there are only differences in density. To challenge this problem, this paper proposes a novel parallel multi-stage (PMS) architecture formed by a low, middle, and high deep convolutional neural network (DCNN) sub-model. PMS model automatically learns representative and discriminative hierarchical features, which include three 512 dimension vectors, respectively, and the final representative feature created by linear connection. PMS model describes a robust feature of aerial image through three stages feature. Unlike previous methods, we only use transfer learning and deep learning methods to obtain more discriminative features from scene images while improving performance. Experimental results demonstrate that the proposed PMS model has a more superior performance than the state-of-the-art methods, obtaining average classification accuracies of 98.81% and 95.56%, respectively, on UC Merced (UCM) and aerial image dataset (AID) benchmark datasets.     

Hyperspectral images classification with convolutional neural network and textural feature using limited training samples

In this letter, a new deep learning framework, which integrates textural features of gray level co-occurrence matrix (GLCM) into convolutional neural networks (CNNs) is proposed for hyperspectral images (HSIs) classification using limited number of labeled samples. The proposed method can be implemented in three steps. Firstly, the GLCM textural features are extracted from the first principal component after the principal components analysis (PCA) transformation. Secondly, a CNN is built to extract the deep spectral features from the original HSIs, and the features are concatenated with the textural features obtained in the first step in a concat layer of CNN. Finally, softmax is employed to generate classification maps at the end of the framework. In this way, the CNN focuses on the learning of spectral features only, and the generated textural features are used directly as one set of features before softmax. These lead to the reduction of the requirements for the size of training samples and the improvement of computing efficiency. The experimental results are presented for three HSIs and compared with several advanced deep learning and spectral-spatial classification techniques. The competitive classification accuracy can be obtained, especially when only a limited number of training samples are available.     

Learning ship width and direction by convolutional neural networks without manual labelling

ABSTRACT

Direction and width prediction have become hot topics in accurate ship location, while the resolution of remote sensing data has been increasing. In the current machine learning, demand for extensive manual labelling has become one of the biggest challenges to further improve tasks such as ship attribute prediction. To cope with this problem, in this paper, we propose a method to predict ship direction and width without manual labelling. To this end, pseudo-label generation approaches have been proposed for transfer learning with convolutional neural network (CNN). Experiments demonstrated that the pre-trained classification CNN features could preserve variational information for such attribute prediction. And the proposed pseudo-labels, which are even with limited qualities, could be efficient to train an accurate ship direction and width prediction model through fault-tolerant training by neural networks.     

A semi-supervised convolutional neural network for hyperspectral image classification

Convolutional neural network (CNN) for hyperspectral image classification can provide excellent performance when the number of labeled samples for training is sufficiently large. Unfortunately, a small number of labeled samples are available for training in hyperspectral images. In this letter, a novel semi-supervised convolutional neural network is proposed for the classification of hyperspectral image. The proposed network can automatically learn features from complex hyperspectral image data structures. Furthermore, skip connection parameters are added between the encoder layer and decoder layer in order to make the network suitable for semi-supervised learning. Semi-supervised method is adopted to solve the problem of limited labeled samples. Finally, the network is trained to simultaneously minimize the sum of supervised and unsupervised cost functions. The proposed network is conducted on a widely used hyperspectral image data. The experimental results demonstrate that the proposed approach provides competitive results to state-of-the-art methods.     

Change detection based on Faster R-CNN for high-resolution remote sensing images

Change detection is of great significance in remote sensing. The advent of high-resolution remote sensing images has greatly increased our ability to monitor land use and land cover changes from space. At the same time, high-resolution remote sensing images present a new challenge over other satellite systems, in which time-consuming and tiresome manual procedures must be needed to identify the land use and land cover changes. In recent years, deep learning (DL) has been widely used in the fields of natural image target detection, speech recognition, face recognition, etc., and has achieved great success. Some scholars have applied DL to remote sensing image classification and change detection, but seldomly to high-resolution remote sensing images change detection. In this letter, faster region-based convolutional neural networks (Faster R-CNN) is applied to the detection of high-resolution remote sensing image change. Compared with several traditional and other DL-based change detection methods, our proposed methods based on Faster R-CNN achieve higher overall accuracy and Kappa coefficient in our experiments. In particular, our methods can reduce a large number of false changes.     

Deep multi-feature fusion network for remote sensing images

ABSTRACT

Learning discriminative and robust features is crucial in remote sensing image processing. Many of the currently used approaches are based on Convolutional Neural Networks (CNNs). However, such approaches may not effectively capture various different semantic objects of remote sensing images. To overcome this limitation, we propose a novel end-to-end deep multi-feature fusion network (DMFN). DMFN combines two different deep architecture branches for feature representations; the global and local branch. The global branch, which consists of three losses, is used to learn discriminative features from the whole image. The local branch is then used in the partitioning of the entire image into multiple strips in order to obtain local features. The two branches are then combined, used to learn fusion feature representations for the image. The proposed method is an end-to-end framework during training. Comprehensive validation experiments on two public datasets indicate that relative to existing deep learning approaches, this strategy is superior for both retrieval and classification tasks.     

Shape and margin-aware lung nodule classification in low-dose CT images via soft activation mapping

A number of studies on lung nodule classification lack clinical/biological interpretations of the features extracted by convolutional neural network (CNN). The methods like class activation mapping (CAM) and gradient-based CAM (Grad-CAM) are tailored for interpreting localization and classification tasks while they ignored fine-grained features. Therefore, CAM and Grad-CAM cannot provide optimal interpretation for lung nodule categorization task in low-dose CT images, in that fine-grained pathological clues like discrete and irregular shape and margins of nodules are capable of enhancing sensitivity and specificity of nodule classification with regards to CNN. In this paper, we first develop a soft activation mapping (SAM) to enable fine-grained lung nodule shape & margin (LNSM) feature analysis with a CNN so that it can access rich discrete features. Secondly, by combining high-level convolutional features with SAM, we further propose a high-level feature enhancement scheme (HESAM) to localize LNSM features. Experiments on the LIDC-IDRI dataset indicate that 1) SAM captures more fine-grained and discrete attention regions than existing methods, 2) HESAM localizes more accurately on LNSM features and obtains the state-of-the-art predictive performance, reducing the false positive rate, and 3) we design and conduct a visually matching experiment which incorporates radiologists study to increase the confidence level of applying our method to clinical diagnosis.