Deep multi-feature fusion network for remote sensing images

ABSTRACT

Learning discriminative and robust features is crucial in remote sensing image processing. Many of the currently used approaches are based on Convolutional Neural Networks (CNNs). However, such approaches may not effectively capture various different semantic objects of remote sensing images. To overcome this limitation, we propose a novel end-to-end deep multi-feature fusion network (DMFN). DMFN combines two different deep architecture branches for feature representations; the global and local branch. The global branch, which consists of three losses, is used to learn discriminative features from the whole image. The local branch is then used in the partitioning of the entire image into multiple strips in order to obtain local features. The two branches are then combined, used to learn fusion feature representations for the image. The proposed method is an end-to-end framework during training. Comprehensive validation experiments on two public datasets indicate that relative to existing deep learning approaches, this strategy is superior for both retrieval and classification tasks.     

A generally applicable noise-estimating method for remote sensing images

Local mean and local standard deviations (LMLSD), which is one of the most widely used methods for estimating noise in remote sensing images, is suitable only for the images with many homogeneous regions. For those composed of heterogeneous features and textures, it may cause overestimation of noise. Edge-extracted local standard deviations (EELSD) method performs better than LMLSD in most instances, but it still cannot work out the accurate noise estimation in most heterogeneous images. Spectral and spatial de-correlation (SSDC) is an effective noise-estimation method for hyperspectral images. However, it cannot be applied to single-band or multispectral images because of the use of pixel spectral information in the calculation process. In this article, a new noise-estimating method for remote sensing images, which is based on the principle of LMLSD and has made improvements in three aspects, is proposed. The new method has been tested with several Airborne Visible Infrared Imaging Spectrometer images with different degrees of uniformity. Compared with LMLSD and EELSD, the results of the improved method are more accurate, stable, and applicable in terms of complex land cover types. Furthermore, in contrast to SSDC, this method is suitable not only for hyperspectral images but also for single-band and multispectral images.     

Change detection based on Faster R-CNN for high-resolution remote sensing images

Change detection is of great significance in remote sensing. The advent of high-resolution remote sensing images has greatly increased our ability to monitor land use and land cover changes from space. At the same time, high-resolution remote sensing images present a new challenge over other satellite systems, in which time-consuming and tiresome manual procedures must be needed to identify the land use and land cover changes. In recent years, deep learning (DL) has been widely used in the fields of natural image target detection, speech recognition, face recognition, etc., and has achieved great success. Some scholars have applied DL to remote sensing image classification and change detection, but seldomly to high-resolution remote sensing images change detection. In this letter, faster region-based convolutional neural networks (Faster R-CNN) is applied to the detection of high-resolution remote sensing image change. Compared with several traditional and other DL-based change detection methods, our proposed methods based on Faster R-CNN achieve higher overall accuracy and Kappa coefficient in our experiments. In particular, our methods can reduce a large number of false changes.     

A novel local variance-based filtering method for denoising remote sensing images

In this paper, a novel filtering method is designed for denoising remote sensing image. Firstly, the image domain of noisy image is partitioned into blocks for estimating the variance of Gaussian white noise. Secondly, based on the fact that the variance of the textural region is always larger than that of the homogeneous region, the noisy image is roughly divided into homogeneous and textural regions. Thirdly, a novel filter is designed and is used to reduce the noises. To this end, adaptive windows with appropriate shape and size are selected for each pixel. With the pixels in the window(s), the noise intensity of the central pixel is estimated and further qualified as a noise level. Based on noise levels, pixel values within the filter window are first updated and then filtered by using the proposed filter. Compared with other filtering methods, better performance is achieved in both noise smoothing and detail conserving.     

Accurate non-maximum suppression for object detection in high-resolution remote sensing images

Non-maximum suppression (NMS) is widely adopted as a post-processing step in the state-of-the-art object detection pipelines to merge the nearby detections around one object. However, its performance is affected by objects that are highly overlapped with each other, and its localization accuracy depends solely on the highest scored detection. To tackle this, an accurate NMS method is proposed in this letter, which gradually merges the highly overlapped detections in an iterative way. In each iteration, detections overlapped with the highest scored one are grouped with a harder threshold to regress for a new proposal, and then the scores within the group are softly suppressed. This process is recursively applied on the remaining detections. The proposed method can not only detect more overlapped objects, but also achieve better object localization accuracy. Experimental results demonstrate that this simple and unsupervised method can gain obvious performance improvement on the majority of classes, compared with the state-of-the-art NMS methods.     

A rate control algorithm for encoding multispectral images from remote sensing satellites

This letter presents a rate allocation algorithm that can be used when the Consultative Committee for Space Data Systems (CCSDS) lossy image compressor is applied to encode multispectral images. This algorithm is a numerical solution to the rate control problem when multiple information sources are compressed independently. Given the rate distortion curves and a total bit rate, the algorithm performs an efficient search to find the best rate distribution among the components that produces the lowest distortion. It is shown that the algorithm introduced here is asymptotically optimal. Furthermore, the relationship between performance and complexity can be easily controlled. The efficiency of the algorithm is tested when CCSDS compressor is used to encode the Karhunen and Loève Transform components of multispectral images. The test is performed for a group of images from different multispectral sensors. The convergence speed and the rate distortion curve are evaluated. Numerical results are compared to the performance of the same compressor when it uses the rate control algorithm based on the Gaussian rate distortion function. It is shown that the proposed algorithm can produce improvements over 2 dB in the Peak Signal-to-Noise Ratio even with a reduced computational complexity.     

Fusion of multiple features to produce a segmentation algorithm for remote sensing images

This letter presents an edge direction adaptive watershed segmentation method for remote sensing images. First, the maximum gradient value among different directions is chosen as the single band gradient value of the pixel, and a compound gradient value is then calculated based on the gradient value in each band. Second, the marker-based watershed segmentation is implemented to produce initial over-segmentation result to avoid under-segmentation. Finally, the adjacent objects with high similarity values are merged to reduce over-segmentation, which improves segmentation accuracy. The performance of the proposed method is validated on two satellite images. Experimental results show that, compared with the multi-resolution segmentation method embedded in the eCognition software and the traditional multi-band watershed segmentation method, the proposed method can decrease over-/under-segmentation and thus produce satisfactory segmentation results.     

Learning a two-stage CNN model for multi-sized building detection in remote sensing images

Though tremendous strides have been made in building recognition, to handle multi-sized buildings is fundamental for all building detection pipelines. We explore the reason of the problem in detecting the multi-sized buildings and find that most convolutional neural network (CNN) based recognition approaches aim to be scale-invariant. The cues for recognizing a 3 pixels tall building are fundamentally different than those for recogjnizing a 300 pixels tall building. To tackle this problem, we design a novel two-stage building detection model which contains the region proposal stage and the classification stage. In the region proposal stage, we propose a novel Multi-size Fusion Region Proposal Network (MFRPN) for extracting the feature of various size building and generating wide size range of region proposals. In the classification stage, a deep CNN model is used to distinguish whether the generated region proposals are building regions or not. In order to achieve better performance, we present an improved block voting algorithm by introducing a dynamic weighting strategy which can obtain a more robust classification result increasing the classification accuracy of the region proposals. We attribute this to robust Experimental results on the challenging VHR dataset indicate that our model has a great performance.     

Evaluating the applicability of BRISK for the geometric registration of remote sensing images

Binary Robust Invariant Scalable Keypoints (BRISK) is one of several relatively new matching algorithms aiming to improve well-established algorithms such as Scale-Invariant Feature Transform (SIFT) or Speeded-Up Robust Features. A detailed evaluation of the BRISK applicability for geometric registration of remote sensing images is performed. As the original algorithm was not developed with a focus on remote sensing image matching, a practical processing chain for the image registration of a newly acquired image with a reference image was developed. This chain also includes a modified Random Sample Consensus outlier removal based on the sensor-model of the to-be-registered image. The presented methodology is evaluated and compared to the SIFT operator in terms of repeatability, accuracy, recall and precision. Our results show that BRISK performs very well on remote sensing images and together with the sensor-model-based outlier removal offers a significant improvement over existing image registration methods such as SIFT.     

Object detection in optical remote sensing images by integrating object-to-object relationships

ABSTRACT

In recent years, deep-learning-based methods for remote sensing image interpretation have undergone rapid development, due to the increasing amount of image data and the advanced techniques of machine learning. The abundant spatial and contextual information within the images is helpful to improve the interpretation performance. However, the contextual information is ignored by most of the current deep-learning-based methods. In this letter, we explore the contextual information by taking advantage of the object-to-object relationship. Then, the feature representation of the individual objects can be enhanced. To be specific, we first build a knowledge database which reveals the relationship between different categories and generate a region-to-region graph that indicates the relationship between different regions of interest (RoIs). For each RoI, the features of its related regions are then combined with the original region features, and the fused features are finally used for object detection. The experiments conducted on a public ten-class object detection dataset demonstrate the validity of the proposed method.     

Spatial contextual classification of remote sensing images using a Gaussian process

Gaussian process (GP) classifiers represent a powerful and interesting theoretical framework for the Bayesian classification of remote sensing images. However, the integration of spatial information in GP classifier is still an open question, while researches have demonstrated that the classification results could be improved when the spatial information is used. In this context, in order to improve the performance of the traditional GP classifier, we propose to use Markov random fields (MRFs) to refine the classification results with the neighbourhood information in the images. In the proposed method (denoted as GP-MRF), the MRF model is used as a post-processing step to the pixelwise results with GP classifier which classifies each pixel in the image separately. Therefore, the proposed GP-MRF approach promotes solutions in which adjacent pixels are likely to belong to the same class. Experimental results show that the GP-MRF could achieve better classification accuracy compared to the original GP classifier and the state-of-the-art spatial contextual classification methods.     

Object recognition in remote sensing images using sparse deep belief networks

Object recognition has been one of the hottest issues in the field of remote sensing image analysis. In this letter, a new pixel-wise learning method based on deep belief networks (DBNs) for object recognition is proposed. The method is divided into two stages, the unsupervised pre-training stage and the supervised fine-tuning stage. Given a training set of images, a pixel-wise unsupervised feature learning algorithm is utilized to train a mixed structural sparse restricted Boltzmann machine (RBM). After that, the outputs of this RBM are put into the next RBM as inputs. By stacking several layers of RBM, the deep generative model of DBNs is built. At the fine-tuning stage, a supervised layer is attached to the top of the DBN and labels of the data are put into this layer. The whole network is then trained using the back-propagation (BP) algorithm with sparse penalty. Finally, the deep model generates good joint distribution of images and their labels. Comparative experiments are conducted on our dataset acquired by QuickBird with 60 cm resolution and the recognition results demonstrate the accuracy and efficiency of our proposed method.     

Vehicle detection in remote sensing images using denoizing-based convolutional neural networks

Vehicle detection in remote sensing images is a tough task and of great significance due to the fast increasing number of vehicles occurring in big cities. Recently, convolutional neural network (CNN)-based methods have achieved excellent performance in classification task due to their powerful abilities in high-level feature extraction. However, overfitting is a serious problem in CNN when applying complicated fully-connected layers, especially when the quantity of training samples is limited. In order to tackle this problem, a denoizing-based CNN called DCNN is proposed in this letter. More specially, a CNN with one fully-connected layer is pre-trained first for feature extraction. After that, features of this fully-connected layer are corrupted and used to pre-train a stacked denoizing autoencoder (SDAE) in an unsupervised way. Then, the pre-trained SDAE is added into the CNN as the fully-connected layer. After fine-tuning, DCNN can make the extracted features more robust and the detecting rate higher. With the help of our proposed locating method, vehicles can be detected effectively even when they are parked in a residential area. Comparative experiments demonstrate that our method has achieved state-of-the-art performance.     

A Y-Net deep learning method for road segmentation using high-resolution visible remote sensing images

Road segmentation from high-resolution visible remote sensing images provides an effective way for automatic road network forming. Recently, deep learning methods based on convolutional neural networks (CNNs) are widely applied in road segmentation. However, it is a challenge for most CNN-based methods to achieve high segmentation accuracy when processing high-resolution visible remote sensing images with rich details. To handle this problem, we propose a road segmentation method based on a Y-shaped convolutional network (indicated as Y-Net). Y-Net contains a two-arm feature extraction module and a fusion module. The feature extraction module includes a deep downsampling-to-upsampling sub-network for semantic features and a convolutional sub-network without downsampling for detail features. The fusion module combines all features for road segmentation. Benefiting from this scheme, the Y-Net can well segment multi-scale roads (both wide and narrow roads) from high-resolution images. The testing and comparative experiments on a public dataset and a private dataset show that Y-Net has higher segmentation accuracy than four other state-of-art methods, FCN (Fully Convolutional Network), U-Net, SegNet, and FC-DenseNet (Fully Convolutional DenseNet). Especially, Y-Net accurately segments contours of narrow roads, which are missed by the comparative methods.     

Mutual information based multi-modal remote sensing image registration using adaptive feature weight

Multi-module images registration is a challenging task in image processing, and more especially in the field of remote sensing. In this letter, we strive to present a novel mutual information scheme for image registration in remote sensing scenario based on feature map technique. We firstly take saliency detection advantages to extract geographic pattern, and then utilize the efficient Laplacian of Gaussian(LOG) and Guided Filter methods to construct a new feature map based on different characteristic of multi-channel images. To avoid practical traps of sub-optimization, we propose an novel mutual information(MI) algorithm based on an adapted weight strategy. The proposed model divides an image into patches and assigns weighted values according to patch similarities in order to solve the optimization problem, improve accuracy and enhance performance. Note that, our proposed method incorporates the LOG and Guided Filter methods into image registration for the first time to construct a new feature map based on differences and similarities strategy. Experiments are conducted over island and coastline scenes, and reveal that our hybrid model has a significant performance and outperforms the state-of-the-art methods in remote sensing image registration.     

Extracting rooftops from remote sensing images using both top-down and bottom-up processes

Building rooftop extraction is one of the most challenging tasks in the field of remote sensing image analysis. Existing methods usually perform poorly due to the complexity of the object and background. In this letter, we propose a novel framework for rooftop localization and geometric structure recovery via combining the strength of top-down and bottom-up methods. Specifically, a novel energy function combining the region term, the shape term, and the penalty term is proposed to eliminate the effect of unclosed contours and other disturbances such as park lots and shadows. In order to take advantage of bottom-up cues, a new penalty term is proposed, in which the position is determined by the directional spatial relationship between building and its shadow, and the orientation of a possible rooftop is estimated by the spatial context. A simulated annealing (SA) algorithm is applied to optimizing the function, which is fused with Markov chain Monte Carlo (MCMC) technique, and special transition kernels are designed in order to achieve convergent extraction results and get rid of local minimum. Experiments on IKONOS images demonstrate the robustness and accuracy of our method.     

Active learning for training sample selection in remote sensing image classification using spatial information

Collection of training samples for remote sensing image classification is always time-consuming and expensive. In this context, active learning (AL) that aims at using limited training samples to achieve promising classification performances is developed. Recently, integration of spatial information into AL exhibits new potential for image classification. In this letter, an AL approach with two-stage spatial computation (AL-2SC) is proposed to improve the selection of training samples. The spatial features derived from remote sensing image and the probability outputs from the neighboring pixels are introduced in AL process. Moreover, we compare several AL approaches which take spatial information into account. In experiments, random sampling (RS) and four AL methods, including AL using breaking ties heuristic (BT), AL with spatial feature (AL-SF), AL with neighbouring responses (AL-NR), and AL-2SC, are considered. Three remote sensing datasets, including one hyperspectral and two multispectral images, are used to compare the performance of different methods. It is illustrated that, the utilization of spatial information is very important for the improvement of AL performance, and the proposed AL-2SC shows the most satisfactory result.     

Combining transductive and active learning to improve object-based classification of remote sensing images

In this letter, we propose a new active transductive learning (ATL) framework for object-based classification of satellite images. The framework couples graph-based label propagation with active learning (AL) to exploit positive aspects of the two learning settings. The transductive approach considers both labelled and unlabelled image objects to perform its classification as they are all available at training time while the AL strategy smartly guides the construction of the training set employed by the learner. The proposed framework was tested in the context of a land cover classification task using RapidEye optical imagery. A reference land cover map was elaborated over the whole study area in order to get reliable information about the performance of the ATL framework. The experimental evaluation underlines that, with a reasonable amount of training data, our framework outperforms state of the art classification methods usually employed in the field of remote sensing.     

An adaptively weighted multi-feature method for object-based change detection in high spatial resolution remote sensing images

ABSTRACT

In this letter, an adaptively weighted multi-feature-based method for unsupervised object-based change detection in high-resolution remote sensing images is proposed. First, a sample selection strategy using fuzzy c-means is designed to obtain high precision pseudo-samples in an unsupervised way. Second, the multiple candidate features are categorized into spectral, geometric and textural groups and two kinds of weights are involved taking into account different contributions. The within-group weights for each feature can be calculated based on single-feature distribution curve without any prior distribution assumption, and the between-group weights for each group are decided by scatter matrices. Third, the weighted multi-feature method is used to generate a reliable difference image which is directly clustered to obtain the final change map. Compared with the other five state-of-the-art methods, the experimental results on two datasets demonstrate the effectiveness and superiority of the proposed method.