Vehicle detection in remote sensing images using denoizing-based convolutional neural networks

Vehicle detection in remote sensing images is a tough task and of great significance due to the fast increasing number of vehicles occurring in big cities. Recently, convolutional neural network (CNN)-based methods have achieved excellent performance in classification task due to their powerful abilities in high-level feature extraction. However, overfitting is a serious problem in CNN when applying complicated fully-connected layers, especially when the quantity of training samples is limited. In order to tackle this problem, a denoizing-based CNN called DCNN is proposed in this letter. More specially, a CNN with one fully-connected layer is pre-trained first for feature extraction. After that, features of this fully-connected layer are corrupted and used to pre-train a stacked denoizing autoencoder (SDAE) in an unsupervised way. Then, the pre-trained SDAE is added into the CNN as the fully-connected layer. After fine-tuning, DCNN can make the extracted features more robust and the detecting rate higher. With the help of our proposed locating method, vehicles can be detected effectively even when they are parked in a residential area. Comparative experiments demonstrate that our method has achieved state-of-the-art performance.     

SatCNN: satellite image dataset classification using agile convolutional neural networks

With the launch of various remote-sensing satellites, more and more high-spatial resolution remote-sensing (HSR-RS) images are becoming available. Scene classification of such a huge volume of HSR-RS images is a big challenge for the efficiency of the feature learning and model training. The deep convolutional neural network (CNN), a typical deep learning model, is an efficient end-to-end deep hierarchical feature learning model that can capture the intrinsic features of input HSR-RS images. However, most published CNN architectures are borrowed from natural scene classification with thousands of training samples, and they are not designed for HSR-RS images. In this paper, we propose an agile CNN architecture, named as SatCNN, for HSR-RS image scene classification. Based on recent improvements to modern CNN architectures, we use more efficient convolutional layers with smaller kernels to build an effective CNN architecture. Experiments on SAT data sets confirmed that SatCNN can quickly and effectively learn robust features to handle the intra-class diversity even with small convolutional kernels, and the deeper convolutional layers allow spontaneous modelling of the relative spatial relationships. With the help of fast graphics processing unit acceleration, SatCNN can be trained within about 40 min, achieving overall accuracies of 99.65% and 99.54%, which is the state-of-the-art for SAT data sets.     

A dense convolutional neural network for hyperspectral image classification

In this letter, a dense convolutional neural network (DCNN) is proposed for hyperspectral image classification, aiming to improve classification performance by promoting feature reuse and strengthening the flow of features and gradients. In the network, features are learned mainly through designed dense blocks, where feature maps generated in each layer can connect directly to the subsequent layers by a concatenation mode. Experiments are conducted on two well-known hyperspectral image data sets, using the proposed method and four comparable methods. Results demonstrate that overall accuracies of the DCNN reached 97.61 and 99.50% for the respective image data sets, representing an obvious improvement over the accuracies of the compared methods. The study confirms that the DCNN can provide more discriminable features for hyperspectral image classification and can offer higher classification accuracies and smoother classification maps.     

A multi-depth convolutional neural network for SAR image classification

The convolutional neural network has been widely used in synthetic aperture radar (SAR) image classification, for it can learn discriminative features from massive amounts of data. However, it is short of distinctive learning mechanisms for different regions in SAR images. In this letter, a novel architecture called multi-depth convolutional neural network (Multi-depth CNN) is proposed which can select different levels of features for classification. Differing from classical convolutional neural network, Multi-depth CNN adopts a piecewise back-propagation method to optimize the network. Meanwhile, compared with classical convolutional neural network, the proposed network can reduce the training time effectively. Experimental results on two datasets demonstrate that the proposed network can achieve better classification accuracy compared with some state-of-art algorithms.     

A semi-supervised convolutional neural network for hyperspectral image classification

Convolutional neural network (CNN) for hyperspectral image classification can provide excellent performance when the number of labeled samples for training is sufficiently large. Unfortunately, a small number of labeled samples are available for training in hyperspectral images. In this letter, a novel semi-supervised convolutional neural network is proposed for the classification of hyperspectral image. The proposed network can automatically learn features from complex hyperspectral image data structures. Furthermore, skip connection parameters are added between the encoder layer and decoder layer in order to make the network suitable for semi-supervised learning. Semi-supervised method is adopted to solve the problem of limited labeled samples. Finally, the network is trained to simultaneously minimize the sum of supervised and unsupervised cost functions. The proposed network is conducted on a widely used hyperspectral image data. The experimental results demonstrate that the proposed approach provides competitive results to state-of-the-art methods.     

A multi-model ensemble method based on convolutional neural networks for aircraft detection in large remote sensing images

Aircraft detection is one of hot issues of remote sensing (RS) image interpretation. Different from existing methods which aim at detecting objects in regions of interest (ROI) of airports, in this work, we propose a novel and effective aircraft detection framework based on convolutional neural networks (CNN) to detect multi-scale targets in extremely large and complicated scenes. In particular, we design a constrained EdgeBoxes (CEdgeBoxes) approach to generate a modest number of target candidates quickly and precisely. Then, in order to overcome the drawbacks of using handcrafted features and traditional classifiers, a modified GoogLeNet combined with Fast Region-based CNN (R-CNN) is designed to extract useful features from RS images and then detect various kinds of multi-scale aircrafts. Most importantly, we propose a new multi-model ensemble method to decrease the false alarm rate (FAR) caused by the imbalanced distribution of complicated targets and backgrounds. Thorough experiments, which are conducted on the dataset acquired by QuickBird with a resolution of 0.6 m, demonstrate our method can effectively detect aircrafts in large scenes with low FARs.     

Using Siamese capsule networks for remote sensing scene classification

ABSTRACT

The convolutional neural network (CNN) is widely used for image classification because of its powerful feature extraction capability. The key challenge of CNN in remote sensing (RS) scene classification is that the size of data set is small and images in each category vary greatly in position and angle, while the spatial information will be lost in the pooling layers of CNN. Consequently, how to extract accurate and effective features is very important. To this end, we present a Siamese capsule network to address these issues. Firstly, we introduce capsules to extract the spatial information of the features so as to learn equivariant representations. Secondly, to improve the classification accuracy of the model on small data sets, the proposed model utilizes the structure of the Siamese network as embedded verification. Finally, the features learned through Capsule networks are regularized by a metric learning term to improve the robustness of our model. The effectiveness of the model on three benchmark RS data sets is verified by different experiments. Experimental results demonstrate that the comprehensive performance of the proposed method surpasses other existing methods.     

A segmented particle swarm optimization convolutional neural network for land cover and land use classification of remote sensing images

ABSTRACT

The development of automatic classification methods in neural networks is an important topic in the field of land cover and land use (LULC) classification of remote sensing images. Here, we proposed a new segmented particle swarm convolutional neural network model (SPSO-CNN) by combining the subsection particle swarm algorithm with a convolutional neural network. The SPSO-CNN was applied to experiment of LULC classification of GF-1 high resolution remote sensing image. The results showed that SPSO-CNN achieved high precision, recall, F1 score and total precision in the LULC classification of remote sensing image with high spatial resolution, demonstrating the advantage and potential of applying SPSO-CNN to the LULC classification of remote sensing images.     

Single image super-resolution for whole slide image using convolutional neural networks and self-supervised color normalization

High-quality whole slide scanners used for animal and human pathology scanning are expensive and can produce massive datasets, which limits the access to and adoption of this technique. As a potential solution to these challenges, we present a deep learning-based approach making use of single image super-resolution (SISR) to reconstruct high-resolution histology images from low-resolution inputs. Such low-resolution images can easily be shared, require less storage, and can be acquired quickly using widely available low-cost slide scanners. The network consists of multi-scale fully convolutional networks capable of capturing hierarchical features. Conditional generative adversarial loss is incorporated to penalize blurriness in the output images. The network is trained using a progressive strategy where the scaling factor is sampled from a normal distribution with an increasing mean. The results are evaluated with quantitative metrics and are used in a clinical histopathology diagnosis procedure which shows that the SISR framework can be used to reconstruct high-resolution images with clinical level quality. We further propose a self-supervised color normalization method that can remove staining variation artifacts. Quantitative evaluations show that the SISR framework can generalize well on unseen data collected from other patient tissue cohorts by incorporating the color normalization method.     

Hyperspectral image classification based on convolutional neural network and random forest

ABSTRACT

Deep learning-based methods, especially deep convolutional neural network (CNN), have proven their powerfulness in hyperspectral image (HSI) classification. On the other hand, ensemble learning is a useful method for classification task. In this letter, in order to further improve the classification accuracy, the combination of CNN and random forest (RF) is proposed for HSI classification. The well-designed CNN is used as individual classifier to extract the discriminant features of HSI and RF randomly selects the extracted features and training samples to formulate a multiple classifier system. Furthermore, the learned weights of CNN are adopted to initialize other individual CNN. Experimental results with two hyperspectral data sets indicate that the proposed method provides competitive classification results compared with state-of-the-art methods.     

Parallel multi-stage features fusion of deep convolutional neural networks for aerial scene classification

Aerial scene classification is a challenging task in the remote sensing image processing field. Owing to some similar scene, there are only differences in density. To challenge this problem, this paper proposes a novel parallel multi-stage (PMS) architecture formed by a low, middle, and high deep convolutional neural network (DCNN) sub-model. PMS model automatically learns representative and discriminative hierarchical features, which include three 512 dimension vectors, respectively, and the final representative feature created by linear connection. PMS model describes a robust feature of aerial image through three stages feature. Unlike previous methods, we only use transfer learning and deep learning methods to obtain more discriminative features from scene images while improving performance. Experimental results demonstrate that the proposed PMS model has a more superior performance than the state-of-the-art methods, obtaining average classification accuracies of 98.81% and 95.56%, respectively, on UC Merced (UCM) and aerial image dataset (AID) benchmark datasets.     

HeadLocNet: Deep convolutional neural networks for accurate classification and multi-landmark localization of head CTs

Cochlear implants (CIs) are used to treat subjects with hearing loss. In a CI surgery, an electrode array is inserted into the cochlea to stimulate auditory nerves. After surgery, CIs need to be programmed. Studies have shown that the cochlea-electrode spatial relationship derived from medical images can guide CI programming and lead to significant improvement in hearing outcomes. We have developed a series of algorithms to segment the inner ear anatomy and localize the electrodes. But, because clinical head CT images are acquired with different protocols, the field of view and orientation of the image volumes vary greatly. As a consequence, visual inspection and manual image registration to an atlas image are needed to document their content and to initialize intensity-based registration algorithms used in our processing pipeline. For large-scale evaluation and deployment of our methods these steps need to be automated. In this article we propose to achieve this with a deep convolutional neural network (CNN) that can be trained end-to-end to classify a head CT image in terms of its content and to localize landmarks. The detected landmarks can then be used to estimate a point-based registration with the atlas image in which the same landmark set's positions are known. We achieve 99.5% classification accuracy and an average localization error of 3.45 mm for 7 landmarks located around each inner ear. This is better than what was achieved with earlier methods we have proposed for the same tasks.     

Spectral–spatial classification of hyperspectral images using deep convolutional neural networks

In this letter, a novel deep learning framework for hyperspectral image classification using both spectral and spatial features is presented. The framework is a hybrid of principal component analysis, deep convolutional neural networks (DCNNs) and logistic regression (LR). The DCNNs for hierarchically extract deep features is introduced into hyperspectral image classification for the first time. The proposed technique consists of two steps. First, feature map generation algorithm is presented to generate the spectral and spatial feature maps. Second, the DCNNs-LR classifier is trained to get useful high-level features and to fine-tune the whole model. Comparative experiments conducted over widely used hyperspectral data indicate that DCNNs-LR classifier built in this proposed deep learning framework provides better classification accuracy than previous hyperspectral classification methods.     

Remote sensing image scene classification based on generative adversarial networks

Scene classification of remote sensing images plays an important role in many remote sensing image applications. Training a good classifier needs a large number of training samples. The labeled samples are often scarce and difficult to obtain, and annotating a large number of samples is time-consuming. We propose a novel remote sensing image scene classification framework based on generative adversarial networks (GAN) in this paper. GAN can improve the generalization ability of machine learning network model. However, generating large-size images, especially high-resolution remote sensing images is difficult. To address this issue, the scaled exponential linear units (SELU) are applied into the GAN to generate high quality remote sensing images. Experiments carried out on two datasets show that our approach can obtain the state-of-the-art results compared with the classification results of the classic deep convolutional neural networks, especially when the number of training samples is small.     

Region-based cascade pooling of convolutional features for HRRS image retrieval

High-resolution remote sensing (HRRS) images contain abundant and complex visual contents. It is very important to extract powerful features to represent the complex contents of HRRS images in the image retrieval. This letter proposes a region-based cascade pooling (RBCP) method to aggregate convolutional features from both the pre-trained and the fine-tuned convolutional neural networks (CNNs). The RBCP method adopts small pooling regions, and first uses max-pooling on the feature maps of the last pooling layer, then employs average-pooling on the max-pooled feature maps. Furthermore, the feature map size is related to the input size, then two kinds of input sizes (required input size and original input size) are compared and analyzed for the RBCP features. The simulation results show that the RBCP features perform better than the features with traditional pooling methods, since multiple patch features can be extracted to describe the details of HRRS images. The RBCP method combines the advantages of max-pooling and average-pooling to extract discriminative features, thus it provides competitive results compared with state-of-the-art methods.     

An improved unsupervised representation learning generative adversarial network for remote sensing image scene classification

ABSTRACT

Unsupervised representation learning plays an important role in remote sensing image applications. Generative adversarial network (GAN) is the most popular unsupervised learning method in recent years. However, due to poor data augmentation, many GAN-based methods are often dif?cult to carry out. In this paper, we propose an improved unsupervised representation learning model called multi-layer feature fusion Wasserstein GAN (MF-WGANs) which considers extracting the feature information for remote sensing scene classification from unlabelled samples. First, we introduced a multi-feature fusion layer behind the discriminator to extract the high-level and mid-level feature information. Second, we combined the loss of multi-feature fusion layer and WGAN-GP to generate more stable and high-quality remote sensing images with a resolution of 256 × 256. Finally, the multi-layer perceptron classifier (MLP-classifier) is used to classify the features extracted from the multi-feature fusion layer and evaluated with the UC Merced Land-Use, AID and NWPU-RESISC45 data sets. Experiments show that MF-WGANs has richer data augmentation and better classification performance than other unsupervised representation learning classification models (e.g., MARTA GANs).     

Active learning for training sample selection in remote sensing image classification using spatial information

Collection of training samples for remote sensing image classification is always time-consuming and expensive. In this context, active learning (AL) that aims at using limited training samples to achieve promising classification performances is developed. Recently, integration of spatial information into AL exhibits new potential for image classification. In this letter, an AL approach with two-stage spatial computation (AL-2SC) is proposed to improve the selection of training samples. The spatial features derived from remote sensing image and the probability outputs from the neighboring pixels are introduced in AL process. Moreover, we compare several AL approaches which take spatial information into account. In experiments, random sampling (RS) and four AL methods, including AL using breaking ties heuristic (BT), AL with spatial feature (AL-SF), AL with neighbouring responses (AL-NR), and AL-2SC, are considered. Three remote sensing datasets, including one hyperspectral and two multispectral images, are used to compare the performance of different methods. It is illustrated that, the utilization of spatial information is very important for the improvement of AL performance, and the proposed AL-2SC shows the most satisfactory result.     

Semi-supervised training of deep convolutional neural networks with heterogeneous data and few local annotations: An experiment on prostate histopathology image classification

Convolutional neural networks (CNNs) are state-of-the-art computer vision techniques for various tasks, particularly for image classification. However, there are domains where the training of classification models that generalize on several datasets is still an open challenge because of the highly heterogeneous data and the lack of large datasets with local annotations of the regions of interest, such as histopathology image analysis. Histopathology concerns the microscopic analysis of tissue specimens processed in glass slides to identify diseases such as cancer. Digital pathology concerns the acquisition, management and automatic analysis of digitized histopathology images that are large, having in the order of ${100}^{\prime }{000}^2$ pixels per image. Digital histopathology images are highly heterogeneous due to the variability of the image acquisition procedures. Creating locally labeled regions (required for the training) is time-consuming and often expensive in the medical field, as physicians usually have to annotate the data. Despite the advances in deep learning, leveraging strongly and weakly annotated datasets to train classification models is still an unsolved problem, mainly when data are very heterogeneous. Large amounts of data are needed to create models that generalize well. This paper presents a novel approach to train CNNs that generalize to heterogeneous datasets originating from various sources and without local annotations. The data analysis pipeline targets Gleason grading on prostate images and includes two models in sequence, following a teacher/student training paradigm. The teacher model (a high-capacity neural network) automatically annotates a set of pseudo-labeled patches used to train the student model (a smaller network). The two models are trained with two different teacher/student approaches: semi-supervised learning and semi-weekly supervised learning. For each of the two approaches, three student training variants are presented. The baseline is provided by training the student model only with the strongly annotated data. Classification performance is evaluated on the student model at the patch level (using the local annotations of the Tissue Micro-Arrays Zurich dataset) and at the global level (using the TCGA-PRAD, The Cancer Genome Atlas-PRostate ADenocarcinoma, whole slide image Gleason score). The teacher/student paradigm allows the models to better generalize on both datasets, despite the inter-dataset heterogeneity and the small number of local annotations used. The classification performance is improved both at the patch-level (up to $\kappa =0.6127\pm 0.0133$ from $\kappa =0.5667\pm 0.0285$), at the TMA core-level (Gleason score) (up to $\kappa =0.7645\pm 0.0231$ from $\kappa =0.7186\pm 0.0306$) and at the WSI-level (Gleason score) (up to $\kappa =0.4529\pm 0.0512$ from $\kappa =0.2293\pm 0.1350$). The results show that with the teacher/student paradigm, it is possible to train models that generalize on datasets from entirely different sources, despite the inter-dataset heterogeneity and the lack of large datasets with local annotations.     

Supervised learning with convolutional neural networks for hyperspectral visualization

ABSTRACT

The Visualization for hyperspectral images is demanded by the ground processing system for quick view and information survey. Previous methods using band selection or dimension reduction fail to produce stable details and good colours as reasonable as corresponding multispectral images. In this paper, a mixed strategy is proposed which utilizes multi-band selection and supervised learning to integrate natural spatial details within hyperspectral bands for visualization. In the training stage, multispectral images are introduced as the reference, which are targeted by convolutional neural networks modelling the conversion from hyperspectral bands to multispectral bands. In order to preserve the colour, the hyperspectral band numbers are recorded for those with high correlation to the reference images, which are put into the network for dimension reduction. The proposed method is tested for the EO-1 Hyperion hyperspectral images with LandSat-8 images as the benchmark. The results are compared with five state-of-the-art algorithms. The comparison results show that the present method has good performance in maintaining both structures and colours.