A multi-model ensemble method based on convolutional neural networks for aircraft detection in large remote sensing images

Aircraft detection is one of hot issues of remote sensing (RS) image interpretation. Different from existing methods which aim at detecting objects in regions of interest (ROI) of airports, in this work, we propose a novel and effective aircraft detection framework based on convolutional neural networks (CNN) to detect multi-scale targets in extremely large and complicated scenes. In particular, we design a constrained EdgeBoxes (CEdgeBoxes) approach to generate a modest number of target candidates quickly and precisely. Then, in order to overcome the drawbacks of using handcrafted features and traditional classifiers, a modified GoogLeNet combined with Fast Region-based CNN (R-CNN) is designed to extract useful features from RS images and then detect various kinds of multi-scale aircrafts. Most importantly, we propose a new multi-model ensemble method to decrease the false alarm rate (FAR) caused by the imbalanced distribution of complicated targets and backgrounds. Thorough experiments, which are conducted on the dataset acquired by QuickBird with a resolution of 0.6 m, demonstrate our method can effectively detect aircrafts in large scenes with low FARs.     

A Y-Net deep learning method for road segmentation using high-resolution visible remote sensing images

Road segmentation from high-resolution visible remote sensing images provides an effective way for automatic road network forming. Recently, deep learning methods based on convolutional neural networks (CNNs) are widely applied in road segmentation. However, it is a challenge for most CNN-based methods to achieve high segmentation accuracy when processing high-resolution visible remote sensing images with rich details. To handle this problem, we propose a road segmentation method based on a Y-shaped convolutional network (indicated as Y-Net). Y-Net contains a two-arm feature extraction module and a fusion module. The feature extraction module includes a deep downsampling-to-upsampling sub-network for semantic features and a convolutional sub-network without downsampling for detail features. The fusion module combines all features for road segmentation. Benefiting from this scheme, the Y-Net can well segment multi-scale roads (both wide and narrow roads) from high-resolution images. The testing and comparative experiments on a public dataset and a private dataset show that Y-Net has higher segmentation accuracy than four other state-of-art methods, FCN (Fully Convolutional Network), U-Net, SegNet, and FC-DenseNet (Fully Convolutional DenseNet). Especially, Y-Net accurately segments contours of narrow roads, which are missed by the comparative methods.     

Aircraft detection in remote sensing images using centre-based proposal regions and invariant features

ABSTRACT

Aircraft detection in remote sensing imagery has drawn much attention in recent years, which plays an important role in various military and civil applications. While many advanced works have been developed with powerful learning algorithms in natural images, there still lacks an effective one to detect aircraft precisely in remote sensing images, especially in some complicated conditions. In this paper, a novel method is designed to detect aircraft precisely, named aircraft detection using Centre-based Proposal regions and Invariant Features (CPIF), which can handle some difficult image deformations, especially rotations. Our framework mainly contains three steps. Firstly, we propose an algorithm to extract proposal regions from remote sensing imagery. Secondly, an ensemble learning classifier with the rotation-invariant HOG is trained for aircraft classification. Lastly, we detect aircraft in remote sensing images by combining the products of the above steps. The proposed method is evaluated on a public dataset RSOD and the results are performed to demonstrate the superiority and effectiveness in comparison with the state-of-the-art methods.     

Application of a parallel spectral–spatial convolution neural network in object-oriented remote sensing land use classification

Due to the abundance of spatial information and relative lack of spectral information in high spatial resolution remote sensing images, a land use classification method for high-resolution remote sensing images is proposed based on a parallel spectral-spatial convolutional neural network (CNN) and object-oriented remote sensing technology. The contour of a remote sensing object is taken as the boundary and the set of pixels that comprises the object are extracted to form the input data set for the deep neural network. The proposed network considers both the features of the object and the pixels which forms the object. The spatial and spectral features of remote sensing image objects are extracted independently in the parallel network using panchromatic and multispectral remote sensing techniques. Then, through a fully connected layer, both spectral and spatial information are integrated to produce remote sensing object class coding. The experimental results demonstrate that the parallel spectral-spatial CNN, which combines spatial and spectral features, achieves better classification performance than the individual CNN. Therefore, the proposed method provides a novel approach to land use classification based on high spatial resolution remote sensing images.     

Change detection based on Faster R-CNN for high-resolution remote sensing images

Change detection is of great significance in remote sensing. The advent of high-resolution remote sensing images has greatly increased our ability to monitor land use and land cover changes from space. At the same time, high-resolution remote sensing images present a new challenge over other satellite systems, in which time-consuming and tiresome manual procedures must be needed to identify the land use and land cover changes. In recent years, deep learning (DL) has been widely used in the fields of natural image target detection, speech recognition, face recognition, etc., and has achieved great success. Some scholars have applied DL to remote sensing image classification and change detection, but seldomly to high-resolution remote sensing images change detection. In this letter, faster region-based convolutional neural networks (Faster R-CNN) is applied to the detection of high-resolution remote sensing image change. Compared with several traditional and other DL-based change detection methods, our proposed methods based on Faster R-CNN achieve higher overall accuracy and Kappa coefficient in our experiments. In particular, our methods can reduce a large number of false changes.     

Tree extraction from multi-scale UAV images using Mask R-CNN with FPN

ABSTRACT

Tree detection and counting have been performed using conventional methods or high costly remote sensing data. In the past few years, deep learning techniques have gained significant progress in the remote sensing area. Namely, convolutional neural networks (CNNs) have been recognized as one of the most successful and widely used deep learning approaches and they have been used for object detection. In this paper, we employed a Mask R-CNN model and feature pyramid network (FPN) for tree extraction from high-resolution RGB unmanned aerial vehicle (UAV) data. The main aim of this paper is to explore the employed method in images with different scales and tree contents. For this purpose, UAV images from two different areas were acquired and three big-scale test images were created for experimental analysis and accuracy assessment. According to the accuracy analyses, despite the scale and the content changes, the proposed model maintains its detection accuracy to a large extent. To our knowledge, this is the first time a Mask R-CNN model with FPN has been used with UAV data for tree extraction.     

Region-based cascade pooling of convolutional features for HRRS image retrieval

High-resolution remote sensing (HRRS) images contain abundant and complex visual contents. It is very important to extract powerful features to represent the complex contents of HRRS images in the image retrieval. This letter proposes a region-based cascade pooling (RBCP) method to aggregate convolutional features from both the pre-trained and the fine-tuned convolutional neural networks (CNNs). The RBCP method adopts small pooling regions, and first uses max-pooling on the feature maps of the last pooling layer, then employs average-pooling on the max-pooled feature maps. Furthermore, the feature map size is related to the input size, then two kinds of input sizes (required input size and original input size) are compared and analyzed for the RBCP features. The simulation results show that the RBCP features perform better than the features with traditional pooling methods, since multiple patch features can be extracted to describe the details of HRRS images. The RBCP method combines the advantages of max-pooling and average-pooling to extract discriminative features, thus it provides competitive results compared with state-of-the-art methods.     

Semantic pixel labelling in remote sensing images using a deep convolutional encoder-decoder model

In this letter, we propose a deep convolutional encoder-decoder model for remote sensing images semantic pixel labelling. Specifically, the encoder network is employed to extract the high-level semantic feature of hyperspectral images and the decoder network is employed to map the low resolution feature maps to full input resolution feature maps for pixel-wise labelling. Different from traditional convolutional layers we use a ‘dilated convolution’ which effectively enlarge the receptive field of filters in order to incorporate more context information. Also the fully connected conditional random field (CRF) is integrated into the model so that the network can be trained end-to-end. CRF can effectively improve the localization performance. Experiments on the Vaihingen and Potsdam dataset demonstrate that our model can make promising performance.     

Vehicle detection in remote sensing images using denoizing-based convolutional neural networks

Vehicle detection in remote sensing images is a tough task and of great significance due to the fast increasing number of vehicles occurring in big cities. Recently, convolutional neural network (CNN)-based methods have achieved excellent performance in classification task due to their powerful abilities in high-level feature extraction. However, overfitting is a serious problem in CNN when applying complicated fully-connected layers, especially when the quantity of training samples is limited. In order to tackle this problem, a denoizing-based CNN called DCNN is proposed in this letter. More specially, a CNN with one fully-connected layer is pre-trained first for feature extraction. After that, features of this fully-connected layer are corrupted and used to pre-train a stacked denoizing autoencoder (SDAE) in an unsupervised way. Then, the pre-trained SDAE is added into the CNN as the fully-connected layer. After fine-tuning, DCNN can make the extracted features more robust and the detecting rate higher. With the help of our proposed locating method, vehicles can be detected effectively even when they are parked in a residential area. Comparative experiments demonstrate that our method has achieved state-of-the-art performance.     

Mutual information based multi-modal remote sensing image registration using adaptive feature weight

Multi-module images registration is a challenging task in image processing, and more especially in the field of remote sensing. In this letter, we strive to present a novel mutual information scheme for image registration in remote sensing scenario based on feature map technique. We firstly take saliency detection advantages to extract geographic pattern, and then utilize the efficient Laplacian of Gaussian(LOG) and Guided Filter methods to construct a new feature map based on different characteristic of multi-channel images. To avoid practical traps of sub-optimization, we propose an novel mutual information(MI) algorithm based on an adapted weight strategy. The proposed model divides an image into patches and assigns weighted values according to patch similarities in order to solve the optimization problem, improve accuracy and enhance performance. Note that, our proposed method incorporates the LOG and Guided Filter methods into image registration for the first time to construct a new feature map based on differences and similarities strategy. Experiments are conducted over island and coastline scenes, and reveal that our hybrid model has a significant performance and outperforms the state-of-the-art methods in remote sensing image registration.     

Scene classification in remote sensing images using a two-stage neural network ensemble model

Scene classification has long been a challenging task in the remote sensing field. Conventional approaches based on hand-crafted features are not suitable in large scale remote sensing images. Convolutional Neural Network (CNN) achieves great success in computer vision field by learning hierarchical features automatically from mass data. However, the shortage of labeled dataset in remote sensing field results in severe overfitting and the ensemble of several networks have better generalization ability than one single network. In this letter, we propose a novel Two-Stage Neural Network Ensemble Model to solve the problems mentioned above. Firstly, to overcome overfitting, we pre-train a CNN using the ImageNet dataset and fine tune the network by labeled remote sensing images. Then, the output features are fed to a Restricted Boltzmann Machine (RBM) Retrained Network to get better feature representations. Finally, in testing stage, a method based on Ensemble Inference Network (EIN) is introduced to enhance the generalization ability by combining the classification results of several networks. Experimental results on the UC Merced Land Use (UCML) dataset demonstrate the effectiveness of our proposed method.     

A segmented particle swarm optimization convolutional neural network for land cover and land use classification of remote sensing images

ABSTRACT

The development of automatic classification methods in neural networks is an important topic in the field of land cover and land use (LULC) classification of remote sensing images. Here, we proposed a new segmented particle swarm convolutional neural network model (SPSO-CNN) by combining the subsection particle swarm algorithm with a convolutional neural network. The SPSO-CNN was applied to experiment of LULC classification of GF-1 high resolution remote sensing image. The results showed that SPSO-CNN achieved high precision, recall, F1 score and total precision in the LULC classification of remote sensing image with high spatial resolution, demonstrating the advantage and potential of applying SPSO-CNN to the LULC classification of remote sensing images.     

Selective convolutional neural networks and cascade classifiers for remote sensing image classification

Training convolutional neural network (CNN) architecture fully, using pretrained CNNs as feature extractors, and fine-tuning pretrained CNNs on target datasets are three popular strategies used in state-of-the-art methods for remote sensing image classification. The full training strategy requires large-scale training dataset, whereas the fine-tuning strategy requires a pretrained model to resume network learning. In this study, we propose a new strategy based on selective CNNs and cascade classifiers to improve the classification accuracy of remote sensing images relative to single CNN. First, we conduct a comparative study of existing pretrained CNNs in terms of data augmentation and the use of fully connected layers. Second, selective CNNs, which based on class separability criterion, are presented to obtain an optimal combination from multiple pretrained models. Finally, classification accuracy is improved by introducing two-stage cascade linear classifiers, the prediction probability of which in the first stage is used as input for the second stage. Experiments on three public remote sensing datasets demonstrate the effectiveness of the proposed method in comparison with state-of-the-art methods.     

ResGANet: Residual group attention network for medical image classification and segmentation

In recent years, deep learning technology has shown superior performance in different fields of medical image analysis. Some deep learning architectures have been proposed and used for computational pathology classification, segmentation, and detection tasks. Due to their simple, modular structure, most downstream applications still use ResNet and its variants as the backbone network. This paper proposes a modular group attention block that can capture feature dependencies in medical images in two independent dimensions: channel and space. By stacking these group attention blocks in ResNet-style, we obtain a new ResNet variant called ResGANet. The stacked ResGANet architecture has 1.51–3.47 times fewer parameters than the original ResNet and can be directly used for downstream medical image segmentation tasks. Many experiments show that the proposed ResGANet is superior to state-of-the-art backbone models in medical image classification tasks. Applying it to different segmentation networks can improve the baseline model in medical image segmentation tasks without changing the network architecture. We hope that this work provides a promising method for enhancing the feature representation of convolutional neural networks (CNNs) in the future.     

Remote sensing image scene classification based on generative adversarial networks

Scene classification of remote sensing images plays an important role in many remote sensing image applications. Training a good classifier needs a large number of training samples. The labeled samples are often scarce and difficult to obtain, and annotating a large number of samples is time-consuming. We propose a novel remote sensing image scene classification framework based on generative adversarial networks (GAN) in this paper. GAN can improve the generalization ability of machine learning network model. However, generating large-size images, especially high-resolution remote sensing images is difficult. To address this issue, the scaled exponential linear units (SELU) are applied into the GAN to generate high quality remote sensing images. Experiments carried out on two datasets show that our approach can obtain the state-of-the-art results compared with the classification results of the classic deep convolutional neural networks, especially when the number of training samples is small.     

Parallel multi-stage features fusion of deep convolutional neural networks for aerial scene classification

Aerial scene classification is a challenging task in the remote sensing image processing field. Owing to some similar scene, there are only differences in density. To challenge this problem, this paper proposes a novel parallel multi-stage (PMS) architecture formed by a low, middle, and high deep convolutional neural network (DCNN) sub-model. PMS model automatically learns representative and discriminative hierarchical features, which include three 512 dimension vectors, respectively, and the final representative feature created by linear connection. PMS model describes a robust feature of aerial image through three stages feature. Unlike previous methods, we only use transfer learning and deep learning methods to obtain more discriminative features from scene images while improving performance. Experimental results demonstrate that the proposed PMS model has a more superior performance than the state-of-the-art methods, obtaining average classification accuracies of 98.81% and 95.56%, respectively, on UC Merced (UCM) and aerial image dataset (AID) benchmark datasets.     

Using Siamese capsule networks for remote sensing scene classification

ABSTRACT

The convolutional neural network (CNN) is widely used for image classification because of its powerful feature extraction capability. The key challenge of CNN in remote sensing (RS) scene classification is that the size of data set is small and images in each category vary greatly in position and angle, while the spatial information will be lost in the pooling layers of CNN. Consequently, how to extract accurate and effective features is very important. To this end, we present a Siamese capsule network to address these issues. Firstly, we introduce capsules to extract the spatial information of the features so as to learn equivariant representations. Secondly, to improve the classification accuracy of the model on small data sets, the proposed model utilizes the structure of the Siamese network as embedded verification. Finally, the features learned through Capsule networks are regularized by a metric learning term to improve the robustness of our model. The effectiveness of the model on three benchmark RS data sets is verified by different experiments. Experimental results demonstrate that the comprehensive performance of the proposed method surpasses other existing methods.     

An improved unsupervised representation learning generative adversarial network for remote sensing image scene classification

ABSTRACT

Unsupervised representation learning plays an important role in remote sensing image applications. Generative adversarial network (GAN) is the most popular unsupervised learning method in recent years. However, due to poor data augmentation, many GAN-based methods are often dif?cult to carry out. In this paper, we propose an improved unsupervised representation learning model called multi-layer feature fusion Wasserstein GAN (MF-WGANs) which considers extracting the feature information for remote sensing scene classification from unlabelled samples. First, we introduced a multi-feature fusion layer behind the discriminator to extract the high-level and mid-level feature information. Second, we combined the loss of multi-feature fusion layer and WGAN-GP to generate more stable and high-quality remote sensing images with a resolution of 256 × 256. Finally, the multi-layer perceptron classifier (MLP-classifier) is used to classify the features extracted from the multi-feature fusion layer and evaluated with the UC Merced Land-Use, AID and NWPU-RESISC45 data sets. Experiments show that MF-WGANs has richer data augmentation and better classification performance than other unsupervised representation learning classification models (e.g., MARTA GANs).     

Object detection in optical remote sensing images by integrating object-to-object relationships

ABSTRACT

In recent years, deep-learning-based methods for remote sensing image interpretation have undergone rapid development, due to the increasing amount of image data and the advanced techniques of machine learning. The abundant spatial and contextual information within the images is helpful to improve the interpretation performance. However, the contextual information is ignored by most of the current deep-learning-based methods. In this letter, we explore the contextual information by taking advantage of the object-to-object relationship. Then, the feature representation of the individual objects can be enhanced. To be specific, we first build a knowledge database which reveals the relationship between different categories and generate a region-to-region graph that indicates the relationship between different regions of interest (RoIs). For each RoI, the features of its related regions are then combined with the original region features, and the fused features are finally used for object detection. The experiments conducted on a public ten-class object detection dataset demonstrate the validity of the proposed method.