Deep Consensus Network: Aggregating predictions to improve object detection in microscopy images

Detection of cells and particles in microscopy images is a common and challenging task. In recent years, detection approaches in computer vision achieved remarkable improvements by leveraging deep learning. Microscopy images pose challenges like small and clustered objects, low signal to noise, and complex shape and appearance, for which current approaches still struggle. We introduce Deep Consensus Network, a new deep neural network for object detection in microscopy images based on object centroids. Our network is trainable end-to-end and comprises a Feature Pyramid Network-based feature extractor, a Centroid Proposal Network, and a layer for ensembling detection hypotheses over all image scales and anchors. We suggest an anchor regularization scheme that favours prior anchors over regressed locations. We also propose a novel loss function based on Normalized Mutual Information to cope with strong class imbalance, which we derive within a Bayesian framework. In addition, we introduce an improved algorithm for Non-Maximum Suppression which significantly reduces the algorithmic complexity. Experiments on synthetic data are performed to provide insights into the properties of the proposed loss function and its robustness. We also applied our method to challenging data from the TUPAC16 mitosis detection challenge and the Particle Tracking Challenge, and achieved results competitive or better than state-of-the-art.     

Object detection in optical remote sensing images by integrating object-to-object relationships

ABSTRACT

In recent years, deep-learning-based methods for remote sensing image interpretation have undergone rapid development, due to the increasing amount of image data and the advanced techniques of machine learning. The abundant spatial and contextual information within the images is helpful to improve the interpretation performance. However, the contextual information is ignored by most of the current deep-learning-based methods. In this letter, we explore the contextual information by taking advantage of the object-to-object relationship. Then, the feature representation of the individual objects can be enhanced. To be specific, we first build a knowledge database which reveals the relationship between different categories and generate a region-to-region graph that indicates the relationship between different regions of interest (RoIs). For each RoI, the features of its related regions are then combined with the original region features, and the fused features are finally used for object detection. The experiments conducted on a public ten-class object detection dataset demonstrate the validity of the proposed method.     

A multi-model ensemble method based on convolutional neural networks for aircraft detection in large remote sensing images

Aircraft detection is one of hot issues of remote sensing (RS) image interpretation. Different from existing methods which aim at detecting objects in regions of interest (ROI) of airports, in this work, we propose a novel and effective aircraft detection framework based on convolutional neural networks (CNN) to detect multi-scale targets in extremely large and complicated scenes. In particular, we design a constrained EdgeBoxes (CEdgeBoxes) approach to generate a modest number of target candidates quickly and precisely. Then, in order to overcome the drawbacks of using handcrafted features and traditional classifiers, a modified GoogLeNet combined with Fast Region-based CNN (R-CNN) is designed to extract useful features from RS images and then detect various kinds of multi-scale aircrafts. Most importantly, we propose a new multi-model ensemble method to decrease the false alarm rate (FAR) caused by the imbalanced distribution of complicated targets and backgrounds. Thorough experiments, which are conducted on the dataset acquired by QuickBird with a resolution of 0.6 m, demonstrate our method can effectively detect aircrafts in large scenes with low FARs.     

Mutual information based multi-modal remote sensing image registration using adaptive feature weight

Multi-module images registration is a challenging task in image processing, and more especially in the field of remote sensing. In this letter, we strive to present a novel mutual information scheme for image registration in remote sensing scenario based on feature map technique. We firstly take saliency detection advantages to extract geographic pattern, and then utilize the efficient Laplacian of Gaussian(LOG) and Guided Filter methods to construct a new feature map based on different characteristic of multi-channel images. To avoid practical traps of sub-optimization, we propose an novel mutual information(MI) algorithm based on an adapted weight strategy. The proposed model divides an image into patches and assigns weighted values according to patch similarities in order to solve the optimization problem, improve accuracy and enhance performance. Note that, our proposed method incorporates the LOG and Guided Filter methods into image registration for the first time to construct a new feature map based on differences and similarities strategy. Experiments are conducted over island and coastline scenes, and reveal that our hybrid model has a significant performance and outperforms the state-of-the-art methods in remote sensing image registration.     

Deep multi-feature fusion network for remote sensing images

ABSTRACT

Learning discriminative and robust features is crucial in remote sensing image processing. Many of the currently used approaches are based on Convolutional Neural Networks (CNNs). However, such approaches may not effectively capture various different semantic objects of remote sensing images. To overcome this limitation, we propose a novel end-to-end deep multi-feature fusion network (DMFN). DMFN combines two different deep architecture branches for feature representations; the global and local branch. The global branch, which consists of three losses, is used to learn discriminative features from the whole image. The local branch is then used in the partitioning of the entire image into multiple strips in order to obtain local features. The two branches are then combined, used to learn fusion feature representations for the image. The proposed method is an end-to-end framework during training. Comprehensive validation experiments on two public datasets indicate that relative to existing deep learning approaches, this strategy is superior for both retrieval and classification tasks.     

DDTNet: A dense dual-task network for tumor-infiltrating lymphocyte detection and segmentation in histopathological images of breast cancer

The morphological evaluation of tumor-infiltrating lymphocytes (TILs) in hematoxylin and eosin (H& E)-stained histopathological images is the key to breast cancer (BCa) diagnosis, prognosis, and therapeutic response prediction. For now, the qualitative assessment of TILs is carried out by pathologists, and computer-aided automatic lymphocyte measurement is still a great challenge because of the small size and complex distribution of lymphocytes. In this paper, we propose a novel dense dual-task network (DDTNet) to simultaneously achieve automatic TIL detection and segmentation in histopathological images. DDTNet consists of a backbone network (i.e., feature pyramid network) for extracting multi-scale morphological characteristics of TILs, a detection module for the localization of TIL centers, and a segmentation module for the delineation of TIL boundaries, where a boundary-aware branch is further used to provide a shape prior to segmentation. An effective feature fusion strategy is utilized to introduce multi-scale features with lymphocyte location information from highly correlated branches for precise segmentation. Experiments on three independent lymphocyte datasets of BCa demonstrate that DDTNet outperforms other advanced methods in detection and segmentation metrics. As part of this work, we also propose a semi-automatic method (TILAnno) to generate high-quality boundary annotations for TILs in H& E-stained histopathological images. TILAnno is used to produce a new lymphocyte dataset that contains 5029 annotated lymphocyte boundaries, which have been released to facilitate computational histopathology in the future.     

Tree extraction from multi-scale UAV images using Mask R-CNN with FPN

ABSTRACT

Tree detection and counting have been performed using conventional methods or high costly remote sensing data. In the past few years, deep learning techniques have gained significant progress in the remote sensing area. Namely, convolutional neural networks (CNNs) have been recognized as one of the most successful and widely used deep learning approaches and they have been used for object detection. In this paper, we employed a Mask R-CNN model and feature pyramid network (FPN) for tree extraction from high-resolution RGB unmanned aerial vehicle (UAV) data. The main aim of this paper is to explore the employed method in images with different scales and tree contents. For this purpose, UAV images from two different areas were acquired and three big-scale test images were created for experimental analysis and accuracy assessment. According to the accuracy analyses, despite the scale and the content changes, the proposed model maintains its detection accuracy to a large extent. To our knowledge, this is the first time a Mask R-CNN model with FPN has been used with UAV data for tree extraction.     

DCAN: Deep contour-aware networks for object instance segmentation from histology images

In histopathological image analysis, the morphology of histological structures, such as glands and nuclei, has been routinely adopted by pathologists to assess the malignancy degree of adenocarcinomas. Accurate detection and segmentation of these objects of interest from histology images is an essential prerequisite to obtain reliable morphological statistics for quantitative diagnosis. While manual annotation is error-prone, time-consuming and operator-dependant, automated detection and segmentation of objects of interest from histology images can be very challenging due to the large appearance variation, existence of strong mimics, and serious degeneration of histological structures. In order to meet these challenges, we propose a novel deep contour-aware network (DCAN) under a unified multi-task learning framework for more accurate detection and segmentation. In the proposed network, multi-level contextual features are explored based on an end-to-end fully convolutional network (FCN) to deal with the large appearance variation. We further propose to employ an auxiliary supervision mechanism to overcome the problem of vanishing gradients when training such a deep network. More importantly, our network can not only output accurate probability maps of histological objects, but also depict clear contours simultaneously for separating clustered object instances, which further boosts the segmentation performance. Our method ranked the first in two histological object segmentation challenges, including 2015 MICCAI Gland Segmentation Challenge and 2015 MICCAI Nuclei Segmentation Challenge. Extensive experiments on these two challenging datasets demonstrate the superior performance of our method, surpassing all the other methods by a significant margin.     

Sketch guided and progressive growing GAN for realistic and editable ultrasound image synthesis

Ultrasound (US) imaging is widely used for anatomical structure inspection in clinical diagnosis. The training of new sonographers and deep learning based algorithms for US image analysis usually requires a large amount of data. However, obtaining and labeling large-scale US imaging data are not easy tasks, especially for diseases with low incidence. Realistic US image synthesis can alleviate this problem to a great extent. In this paper, we propose a generative adversarial network (GAN) based image synthesis framework. Our main contributions include: (1) we present the first work that can synthesize realistic B-mode US images with high-resolution and customized texture editing features; (2) to enhance structural details of generated images, we propose to introduce auxiliary sketch guidance into a conditional GAN. We superpose the edge sketch onto the object mask and use the composite mask as the network input; (3) to generate high-resolution US images, we adopt a progressive training strategy to gradually generate high-resolution images from low-resolution images. In addition, a feature loss is proposed to minimize the difference of high-level features between the generated and real images, which further improves the quality of generated images; (4) the proposed US image synthesis method is quite universal and can also be generalized to the US images of other anatomical structures besides the three ones tested in our study (lung, hip joint, and ovary); (5) extensive experiments on three large US image datasets are conducted to validate our method. Ablation studies, customized texture editing, user studies, and segmentation tests demonstrate promising results of our method in synthesizing realistic US images.     

Aircraft detection in remote sensing images using centre-based proposal regions and invariant features

ABSTRACT

Aircraft detection in remote sensing imagery has drawn much attention in recent years, which plays an important role in various military and civil applications. While many advanced works have been developed with powerful learning algorithms in natural images, there still lacks an effective one to detect aircraft precisely in remote sensing images, especially in some complicated conditions. In this paper, a novel method is designed to detect aircraft precisely, named aircraft detection using Centre-based Proposal regions and Invariant Features (CPIF), which can handle some difficult image deformations, especially rotations. Our framework mainly contains three steps. Firstly, we propose an algorithm to extract proposal regions from remote sensing imagery. Secondly, an ensemble learning classifier with the rotation-invariant HOG is trained for aircraft classification. Lastly, we detect aircraft in remote sensing images by combining the products of the above steps. The proposed method is evaluated on a public dataset RSOD and the results are performed to demonstrate the superiority and effectiveness in comparison with the state-of-the-art methods.     

Using Siamese capsule networks for remote sensing scene classification

ABSTRACT

The convolutional neural network (CNN) is widely used for image classification because of its powerful feature extraction capability. The key challenge of CNN in remote sensing (RS) scene classification is that the size of data set is small and images in each category vary greatly in position and angle, while the spatial information will be lost in the pooling layers of CNN. Consequently, how to extract accurate and effective features is very important. To this end, we present a Siamese capsule network to address these issues. Firstly, we introduce capsules to extract the spatial information of the features so as to learn equivariant representations. Secondly, to improve the classification accuracy of the model on small data sets, the proposed model utilizes the structure of the Siamese network as embedded verification. Finally, the features learned through Capsule networks are regularized by a metric learning term to improve the robustness of our model. The effectiveness of the model on three benchmark RS data sets is verified by different experiments. Experimental results demonstrate that the comprehensive performance of the proposed method surpasses other existing methods.     

A hybrid neural network for hyperspectral image classification

ABSTRACT

Recent research shows that deep learning-based methods can achieve promissing performance when applied to hyperspectral image (HSI) classification in remote sensing, some challenging issues still exist. For example, after a number of 2D convolutions, each feature map may only correspond to a unique dimension of the hyperspectral image. As a result, the relationship between different feature maps from multiple dimensional hyperspectral image can not be extracted well. Another issue is information in extracted feature maps may be erased by pooling operations. To address these problems, we propose a novel hybrid neural network (HNN) for hyperspectral image classification. The HNN uses a multi-branch architecture to extract hyperspectral image features in order to improve its prediction accuracy. Moreover, we build a deconvolution structure to recover the lost information in the pooling operation. In addition, to improve convergence and prevent overfitting, the HNN applies batch normalization (BN) and parametric rectified linear units (PReLU). In the experiments, two public benchmark HSIs are utilized to evaluate the performance of the proposed method. The experimental results demonstrate the superiority of HNN over several well-known methods.     

Object recognition in medical images via anatomy-guided deep learning

PurposeDespite advances in deep learning, robust medical image segmentation in the presence of artifacts, pathology, and other imaging shortcomings has remained a challenge. In this paper, we demonstrate that by synergistically marrying the unmatched strengths of high-level human knowledge (i.e., natural intelligence (NI)) with the capabilities of deep learning (DL) networks (i.e., artificial intelligence (AI)) in garnering intricate details, these challenges can be significantly overcome. Focusing on the object recognition task, we formulate an anatomy-guided deep learning object recognition approach named AAR-DL which combines an advanced anatomy-modeling strategy, model-based non-deep-learning object recognition, and deep learning object detection networks to achieve expert human-like performance.MethodsThe AAR-DL approach consists of 4 key modules wherein prior knowledge (NI) is made use of judiciously at every stage. In the first module AAR-R, objects are recognized based on a previously created fuzzy anatomy model of the body region with all its organs following the automatic anatomy recognition (AAR) approach wherein high-level human anatomic knowledge is precisely codified. This module is purely model-based with no DL involvement. Although the AAR-R operation lacks accuracy, it is robust to artifacts and deviations (much like NI), and provides the much-needed anatomic guidance in the form of rough regions-of-interest (ROIs) for the following DL modules. The 2nd module DL-R makes use of the ROI information to limit the search region to just where each object is most likely to reside and performs DL-based detection of the 2D bounding boxes (BBs) in slices. The 2D BBs hug the shape of the 3D object much better than 3D BBs and their detection is feasible only due to anatomy guidance from AAR-R. In the 3rd module, the AAR model is deformed via the found 2D BBs providing refined model information which now embodies both NI and AI decisions. The refined AAR model more actively guides the 4th refined DL-R module to perform final object detection via DL. Anatomy knowledge is made use of in designing the DL networks wherein spatially sparse objects and non-sparse objects are handled differently to provide the required level of attention for each.ResultsUtilizing 150 thoracic and 225 head and neck (H&N) computed tomography (CT) data sets of cancer patients undergoing routine radiation therapy planning, the recognition performance of the AAR-DL approach is evaluated on 10 thoracic and 16 H&N organs in comparison to pure model-based approach (AAR-R) and pure DL approach without anatomy guidance. Recognition accuracy is assessed via location error/ centroid distance error, scale or size error, and wall distance error. The results demonstrate how the errors are gradually and systematically reduced from the 1st module to the 4th module as high-level knowledge is infused via NI at various stages into the processing pipeline. This improvement is especially dramatic for sparse and artifact-prone challenging objects, achieving a location error over all objects of 4.4 mm and 4.3 mm for the two body regions, respectively. The pure DL approach failed on several very challenging sparse objects while AAR-DL achieved accurate recognition, almost matching human performance, showing the importance of anatomy guidance for robust operation. Anatomy guidance also reduces the time required for training DL networks considerably.Conclusions(i) High-level anatomy guidance improves recognition performance of DL methods. (ii) This improvement is especially noteworthy for spatially sparse, low-contrast, inconspicuous, and artifact-prone objects. (iii) Once anatomy guidance is provided, 3D objects can be detected much more accurately via 2D BBs than 3D BBs and the 2D BBs represent object containment with much more specificity. (iv) Anatomy guidance brings stability and robustness to DL approaches for object localization. (v) The training time can be greatly reduced by making use of anatomy guidance.     

Object representations in ventral and dorsal visual streams: fMRI repetition effects depend on attention and part-whole configuration

The effects of attention and object configuration on the neural responses to short-lag visual image repetition were investigated with fMRI. Attention to one of two object images in a prime display was cued spatially. The images were either intact or split vertically; a manipulation that negates the influence of view-based representations. A subsequent single intact probe image was named covertly. Behavioural priming observed as faster button presses was found for attended primes in both intact and split configurations, but only for uncued primes in the intact configuration. In a voxel-wise analysis, fMRI repetition suppression (RS) was observed in a left mid-fusiform region for attended primes, both intact and split, whilst a right intraparietal region showed repetition enhancement (RE) for intact primes, regardless of attention. In a factorial analysis across regions of interest (ROIs) defined from independent localiser contrasts, RS for attended objects in the ventral stream was significantly left-lateralised, whilst repetition effects in ventral and dorsal ROIs correlated with the amount of priming in specific conditions. These fMRI results extend hybrid theories of object recognition, implicating left ventral stream regions in analytic processing (requiring attention), consistent with prior hypotheses about hemispheric specialisation, and implicating dorsal stream regions in holistic processing (independent of attention).     

Disease quantification on PET/CT images without explicit object delineation

PurposeThe derivation of quantitative information from images in a clinically practical way continues to face a major hurdle because of image segmentation challenges. This paper presents a novel approach, called automatic anatomy recognition-disease quantification (AAR-DQ), for disease quantification (DQ) on positron emission tomography/computed tomography (PET/CT) images. This approach explores how to decouple DQ methods from explicit dependence on object (e.g., organ) delineation through the use of only object recognition results from our recently developed automatic anatomy recognition (AAR) method to quantify disease burden.MethodThe AAR-DQ process starts off with the AAR approach for modeling anatomy and automatically recognizing objects on low-dose CT images of PET/CT acquisitions. It incorporates novel aspects of model building that relate to finding an optimal disease map for each organ. The parameters of the disease map are estimated from a set of training image data sets including normal subjects and patients with metastatic cancer. The result of recognition for an object on a patient image is the location of a fuzzy model for the object which is optimally adjusted for the image. The model is used as a fuzzy mask on the PET image for estimating a fuzzy disease map for the specific patient and subsequently for quantifying disease based on this map. This process handles blur arising in PET images from partial volume effect entirely through accurate fuzzy mapping to account for heterogeneity and gradation of disease content at the voxel level without explicitly performing correction for the partial volume effect. Disease quantification is performed from the fuzzy disease map in terms of total lesion glycolysis (TLG) and standardized uptake value (SUV) statistics. We also demonstrate that the method of disease quantification is applicable even when the “object” of interest is recognized manually with a simple and quick action such as interactively specifying a 3D box ROI. Depending on the degree of automaticity for object and lesion recognition on PET/CT, DQ can be performed at the object level either semi-automatically (DQ-MO) or automatically (DQ-AO), or at the lesion level either semi-automatically (DQ-ML) or automatically.ResultsWe utilized 67 data sets in total: 16 normal data sets used for model building, and 20 phantom data sets plus 31 patient data sets (with various types of metastatic cancer) used for testing the three methods DQ-AO, DQ-MO, and DQ-ML. The parameters of the disease map were estimated using the leave-one-out strategy. The organs of focus were left and right lungs and liver, and the disease quantities measured were TLG, SUV_Mean, and SUV_Max. On phantom data sets, overall error for the three parameters were approximately 6%, 3%, and 0%, respectively, with TLG error varying from 2% for large “lesions” (37 mm diameter) to 37% for small “lesions” (10 mm diameter). On patient data sets, for non-conspicuous lesions, those overall errors were approximately 19%, 14% and 0%; for conspicuous lesions, these overall errors were approximately 9%, 7%, 0%, respectively, with errors in estimation being generally smaller for liver than for lungs, although without statistical significance.ConclusionsAccurate disease quantification on PET/CT images without performing explicit delineation of lesions is feasible following object recognition. Method DQ-MO generally yields more accurate results than DQ-AO although the difference is statistically not significant. Compared to current methods from the literature, almost all of which focus only on lesion-level DQ and not organ-level DQ, our results were comparable for large lesions and were superior for smaller lesions, with less demand on training data and computational resources. DQ-AO and even DQ-MO seem to have the potential for quantifying disease burden body-wide routinely via the AAR-DQ approach.     

Triple U-net: Hematoxylin-aware nuclei segmentation with progressive dense feature aggregation

Nuclei segmentation is a vital step for pathological cancer research. It is still an open problem due to some difficulties, such as color inconsistency introduced by non-uniform manual operations, blurry tumor nucleus boundaries and overlapping tumor cells. In this paper, we aim to leverage the unique optical characteristic of H&E staining images that hematoxylin always stains cell nuclei blue, and eosin always stains the extracellular matrix and cytoplasm pink. Therefore, we extract the Hematoxylin component from RGB images by Beer-Lambert’s Law. According to the optical attribute, the extracted Hematoxylin component is robust to color inconsistency. With the Hematoxylin component, we propose a Hematoxylin-aware CNN model for nuclei segmentation without the necessity of color normalization. Our proposed network is formulated as a Triple U-net structure which includes an RGB branch, a Hematoxylin branch and a Segmentation branch. Then we propose a novel feature aggregation strategy to allow the network to fuse features progressively and to learn better feature representations from different branches. Extensive experiments are performed to qualitatively and quantitatively evaluate the effectiveness of our proposed method. In the meanwhile, it outperforms state-of-the-art methods on three different nuclei segmentation datasets.     

HAL-IA: A Hybrid Active Learning framework using Interactive Annotation for medical image segmentation

High performance of deep learning models on medical image segmentation greatly relies on large amount of pixel-wise annotated data, yet annotations are costly to collect. How to obtain high accuracy segmentation labels of medical images with limited cost (e.g. time) becomes an urgent problem. Active learning can reduce the annotation cost of image segmentation, but it faces three challenges: the cold start problem, an effective sample selection strategy for segmentation task and the burden of manual annotation. In this work, we propose a Hybrid Active Learning framework using Interactive Annotation (HAL-IA) for medical image segmentation, which reduces the annotation cost both in decreasing the amount of the annotated images and simplifying the annotation process. Specifically, we propose a novel hybrid sample selection strategy to select the most valuable samples for segmentation model performance improvement. This strategy combines pixel entropy, regional consistency and image diversity to ensure that the selected samples have high uncertainty and diversity. In addition, we propose a warm-start initialization strategy to build the initial annotated dataset to avoid the cold-start problem. To simplify the manual annotation process, we propose an interactive annotation module with suggested superpixels to obtain pixel-wise label with several clicks. We validate our proposed framework with extensive segmentation experiments on four medical image datasets. Experimental results showed that the proposed framework achieves high accuracy pixel-wise annotations and models with less labeled data and fewer interactions, outperforming other state-of-the-art methods. Our method can help physicians efficiently obtain accurate medical image segmentation results for clinical analysis and diagnosis.     

An adaptively weighted multi-feature method for object-based change detection in high spatial resolution remote sensing images

ABSTRACT

In this letter, an adaptively weighted multi-feature-based method for unsupervised object-based change detection in high-resolution remote sensing images is proposed. First, a sample selection strategy using fuzzy c-means is designed to obtain high precision pseudo-samples in an unsupervised way. Second, the multiple candidate features are categorized into spectral, geometric and textural groups and two kinds of weights are involved taking into account different contributions. The within-group weights for each feature can be calculated based on single-feature distribution curve without any prior distribution assumption, and the between-group weights for each group are decided by scatter matrices. Third, the weighted multi-feature method is used to generate a reliable difference image which is directly clustered to obtain the final change map. Compared with the other five state-of-the-art methods, the experimental results on two datasets demonstrate the effectiveness and superiority of the proposed method.