High resolution histopathology image generation and segmentation through adversarial training

Semantic segmentation of histopathology images can be a vital aspect of computer-aided diagnosis, and deep learning models have been effectively applied to this task with varying levels of success. However, their impact has been limited due to the small size of fully annotated datasets. Data augmentation is one avenue to address this limitation. Generative Adversarial Networks (GANs) have shown promise in this respect, but previous work has focused mostly on classification tasks applied to MR and CT images, both of which have lower resolution and scale than histopathology images. There is limited research that applies GANs as a data augmentation approach for large-scale image semantic segmentation, which requires high-quality image-mask pairs. In this work, we propose a multi-scale conditional GAN for high-resolution, large-scale histopathology image generation and segmentation. Our model consists of a pyramid of GAN structures, each responsible for generating and segmenting images at a different scale. Using semantic masks, the generative component of our model is able to synthesize histopathology images that are visually realistic. We demonstrate that these synthesized images along with their masks can be used to boost segmentation performance, especially in the semi-supervised scenario.     

Towards annotation-efficient segmentation via image-to-image translation

An important challenge and limiting factor in deep learning methods for medical imaging segmentation is the lack of available of annotated data to properly train models. For the specific task of tumor segmentation, the process entails clinicians labeling every slice of volumetric scans for every patient, which becomes prohibitive at the scale of datasets required to train neural networks to optimal performance. To address this, we propose a novel semi-supervised framework that allows training any segmentation (encoder–decoder) model using only information readily available in radiological data, namely the presence of a tumor in the image, in addition to a few annotated images. Specifically, we conjecture that a generative model performing domain translation on this weak label — healthy vs diseased scans — helps achieve tumor segmentation. The proposed GenSeg method first disentangles tumoral tissue from healthy “background” tissue. The latent representation is separated into (1) the common background information across both domains, and (2) the unique tumoral information. GenSeg then achieves diseased-to-healthy image translation by decoding a healthy version of the image from just the common representation, as well as a residual image that allows adding back the tumors. The same decoder that produces this residual tumor image, also outputs a tumor segmentation. Implicit data augmentation is achieved by re-using the same framework for healthy-to-diseased image translation, where a residual tumor image is produced from a prior distribution. By performing both image translation and segmentation simultaneously, GenSeg allows training on only partially annotated datasets. To test the framework, we trained U-Net-like architectures using GenSeg and evaluated their performance on 3 variants of a synthetic task, as well as on 2 benchmark datasets: brain tumor segmentation in MRI (derived from BraTS) and liver metastasis segmentation in CT (derived from LiTS). Our method outperforms the baseline semi-supervised (autoencoder and mean teacher) and supervised segmentation methods, with improvements ranging between 8–14% Dice score on the brain task and 5–8% on the liver task, when only 1% of the training images were annotated. These results show the proposed framework is ideal at addressing the problem of training deep segmentation models when a large portion of the available data is unlabeled and unpaired, a common issue in tumor segmentation.     

Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentation

Supervised deep learning-based methods yield accurate results for medical image segmentation. However, they require large labeled datasets for this, and obtaining them is a laborious task that requires clinical expertise. Semi/self-supervised learning-based approaches address this limitation by exploiting unlabeled data along with limited annotated data. Recent self-supervised learning methods use contrastive loss to learn good global level representations from unlabeled images and achieve high performance in classification tasks on popular natural image datasets like ImageNet. In pixel-level prediction tasks such as segmentation, it is crucial to also learn good local level representations along with global representations to achieve better accuracy. However, the impact of the existing local contrastive loss-based methods remains limited for learning good local representations because similar and dissimilar local regions are defined based on random augmentations and spatial proximity; not based on the semantic label of local regions due to lack of large-scale expert annotations in the semi/self-supervised setting. In this paper, we propose a local contrastive loss to learn good pixel level features useful for segmentation by exploiting semantic label information obtained from pseudo-labels of unlabeled images alongside limited annotated images with ground truth (GT) labels. In particular, we define the proposed contrastive loss to encourage similar representations for the pixels that have the same pseudo-label/GT label while being dissimilar to the representation of pixels with different pseudo-label/GT label in the dataset. We perform pseudo-label based self-training and train the network by jointly optimizing the proposed contrastive loss on both labeled and unlabeled sets and segmentation loss on only the limited labeled set. We evaluated the proposed approach on three public medical datasets of cardiac and prostate anatomies, and obtain high segmentation performance with a limited labeled set of one or two 3D volumes. Extensive comparisons with the state-of-the-art semi-supervised and data augmentation methods and concurrent contrastive learning methods demonstrate the substantial improvement achieved by the proposed method. The code is made publicly available at https://github.com/krishnabits001/pseudo_label_contrastive_training.     

Suggestive annotation of brain MR images with gradient-guided sampling

Machine learning has been widely adopted for medical image analysis in recent years given its promising performance in image segmentation and classification tasks. The success of machine learning, in particular supervised learning, depends on the availability of manually annotated datasets. For medical imaging applications, such annotated datasets are not easy to acquire, it takes a substantial amount of time and resource to curate an annotated medical image set. In this paper, we propose an efficient annotation framework for brain MR images that can suggest informative sample images for human experts to annotate. We evaluate the framework on two different brain image analysis tasks, namely brain tumour segmentation and whole brain segmentation. Experiments show that for brain tumour segmentation task on the BraTS 2019 dataset, training a segmentation model with only 7% suggestively annotated image samples can achieve a performance comparable to that of training on the full dataset. For whole brain segmentation on the MALC dataset, training with 42% suggestively annotated image samples can achieve a comparable performance to training on the full dataset. The proposed framework demonstrates a promising way to save manual annotation cost and improve data efficiency in medical imaging applications.     

Graph temporal ensembling based semi-supervised convolutional neural network with noisy labels for histopathology image analysis

Although convolutional neural networks have achieved tremendous success on histopathology image classification, they usually require large-scale clean annotated data and are sensitive to noisy labels. Unfortunately, labeling large-scale images is laborious, expensive and lowly reliable for pathologists. To address these problems, in this paper, we propose a novel self-ensembling based deep architecture to leverage the semantic information of annotated images and explore the information hidden in unlabeled data, and meanwhile being robust to noisy labels. Specifically, the proposed architecture first creates ensemble targets for feature and label predictions of training samples, by using exponential moving average (EMA) to aggregate feature and label predictions within multiple previous training epochs. Then, the ensemble targets within the same class are mapped into a cluster so that they are further enhanced. Next, a consistency cost is utilized to form consensus predictions under different configurations. Finally, we validate the proposed method with extensive experiments on lung and breast cancer datasets that contain thousands of images. It can achieve 90.5% and 89.5% image classification accuracy using only 20% labeled patients on the two datasets, respectively. This performance is comparable to that of the baseline method with all labeled patients. Experiments also demonstrate its robustness to small percentage of noisy labels.     

Remote sensing image classification using the spatial information obtained by statistical region merging

Considering the characteristics of remote sensing images, incorporating spatial information into traditional pattern recognition technology can improve remote sensing image classification performance. We utilized statistical region merging (SRM) to obtain the spatial similarity between pixels. Then we combined the spatial similarity with the traditional spectral similarity to redefine the distance between pixels, getting a hybrid distance measure. To verify the effect of the spatial information obtained by SRM on remote sensing image classification, we applied the hybrid spatial and spectral distance to the two classifiers based on distance: the optimum-path forest (OPF) and the k-nearest neighbours (k-NN). Therefore, we constructed two contextual classifiers: OPF-SRM and k-NN-SRM. The experimental results on four real land cover images demonstrated the validity of the proposed measure of spatial information since OPF-SRM and k-NN-SRM outperformed the original classifiers and other competitive contextual classifiers.     

Wavelet optimization for content-based image retrieval in medical databases

We propose in this article a content-based image retrieval (CBIR) method for diagnosis aid in medical fields. In the proposed system, images are indexed in a generic fashion, without extracting domain-specific features: a signature is built for each image from its wavelet transform. These image signatures characterize the distribution of wavelet coefficients in each subband of the decomposition. A distance measure is then defined to compare two image signatures and thus retrieve the most similar images in a database when a query image is submitted by a physician. To retrieve relevant images from a medical database, the signatures and the distance measure must be related to the medical interpretation of images. As a consequence, we introduce several degrees of freedom in the system so that it can be tuned to any pathology and image modality. In particular, we propose to adapt the wavelet basis, within the lifting scheme framework, and to use a custom decomposition scheme. Weights are also introduced between subbands. All these parameters are tuned by an optimization procedure, using the medical grading of each image in the database to define a performance measure. The system is assessed on two medical image databases: one for diabetic retinopathy follow up and one for screening mammography, as well as a general purpose database. Results are promising: a mean precision of 56.50%, 70.91% and 96.10% is achieved for these three databases, when five images are returned by the system.     

Semi-supervised mp-MRI data synthesis with StitchLayer and auxiliary distance maximization

The availability of a large amount of annotated data is critical for many medical image analysis applications, in particular for those relying on deep learning methods which are known to be data-hungry. However, annotated medical data, especially multimodal data, is often scarce and costly to obtain. In this paper, we address the problem of synthesizing multi-parameter magnetic resonance imaging data (i.e. mp-MRI), which typically consists of Apparent Diffusion Coefficient (ADC) and T2-weighted (T2w) images, containing clinically significant (CS) prostate cancer (PCa) via semi-supervised learning and adversarial learning. Specifically, our synthesizer generates mp-MRI data in a sequential manner: first utilizing a decoder to generate an ADC map from a 128-d latent vector, followed by translating the ADC to the T2w image via U-Net. The synthesizer is trained in a semi-supervised manner. In the supervised training process, a limited amount of paired ADC-T2w images and the corresponding ADC encodings are provided and the synthesizer learns the paired relationship by explicitly minimizing the reconstruction losses between synthetic and real images. To avoid overfitting limited ADC encodings, an unlimited amount of random latent vectors and unpaired ADC-T2w Images are utilized in the unsupervised training process for learning the marginal image distributions of real images. To improve the robustness for training the synthesizer, we decompose the difficult task of generating full-size images into several simpler tasks which generate sub-images only. A StitchLayer is then employed to seamlessly fuse sub-images together in an interlaced manner into a full-size image. In addition, to enforce the synthetic images to indeed contain distinguishable CS PCa lesions, we propose to also maximize an auxiliary distance of Jensen-Shannon divergence (JSD) between CS and nonCS images. Experimental results show that our method can effectively synthesize a large variety of mp-MRI images which contain meaningful CS PCa lesions, display a good visual quality and have the correct paired relationship between the two modalities of a pair. Compared to the state-of-the-art methods based on adversarial learning (Liu and Tuzel, 2016; Costa et al., 2017), our method achieves a significant improvement in terms of both visual quality and several popular quantitative evaluation metrics.     

Learning to segment images with classification labels

Two of the most common tasks in medical imaging are classification and segmentation. Either task requires labeled data annotated by experts, which is scarce and expensive to collect. Annotating data for segmentation is generally considered to be more laborious as the annotator has to draw around the boundaries of regions of interest, as opposed to assigning image patches a class label. Furthermore, in tasks such as breast cancer histopathology, any realistic clinical application often includes working with whole slide images, whereas most publicly available training data are in the form of image patches, which are given a class label. We propose an architecture that can alleviate the requirements for segmentation-level ground truth by making use of image-level labels to reduce the amount of time spent on data curation. In addition, this architecture can help unlock the potential of previously acquired image-level datasets on segmentation tasks by annotating a small number of regions of interest. In our experiments, we show using only one segmentation-level annotation per class, we can achieve performance comparable to a fully annotated dataset.     

Chest wall segmentation in automated 3D breast ultrasound scans

In this paper, we present an automatic method to segment the chest wall in automated 3D breast ultrasound images. Determining the location of the chest wall in automated 3D breast ultrasound images is necessary in computer-aided detection systems to remove automatically detected cancer candidates beyond the chest wall and it can be of great help for inter- and intra-modal image registration. We show that the visible part of the chest wall in an automated 3D breast ultrasound image can be accurately modeled by a cylinder. We fit the surface of our cylinder model to a set of automatically detected rib-surface points. The detection of the rib-surface points is done by a classifier using features representing local image intensity patterns and presence of rib shadows. Due to attenuation of the ultrasound signal, a clear shadow is visible behind the ribs. Evaluation of our segmentation method is done by computing the distance of manually annotated rib points to the surface of the automatically detected chest wall. We examined the performance on images obtained with the two most common 3D breast ultrasound devices in the market. In a dataset of 142 images, the average mean distance of the annotated points to the segmented chest wall was 5.59 ± 3.08 mm.     

Large-Scale medical image analytics: Recent methodologies,applications and Future directions

Despite the ever-increasing amount and complexity of annotated medical image data, the development of large-scale medical image analysis algorithms has not kept pace with the need for methods that bridge the semantic gap between images and diagnoses. The goal of this position paper is to discuss and explore innovative and large-scale data science techniques in medical image analytics, which will benefit clinical decision-making and facilitate efficient medical data management. Particularly, we advocate that the scale of image retrieval systems should be significantly increased at which interactive systems can be effective for knowledge discovery in potentially large databases of medical images. For clinical relevance, such systems should return results in real-time, incorporate expert feedback, and be able to cope with the size, quality, and variety of the medical images and their associated metadata for a particular domain. The design, development, and testing of the such framework can significantly impact interactive mining in medical image databases that are growing rapidly in size and complexity and enable novel methods of analysis at much larger scales in an efficient, integrated fashion.     

Speckle reduction in ultrasound images using nonisotropic adaptive filtering

In this article, a speckle reduction approach for ultrasound imaging that preserves important features such as edges, corners and point targets is presented. Speckle reduction is an important problem in coherent imaging, such as ultrasound imaging or synthetic aperture radar, and many speckle reduction algorithms have been developed. Speckle is a non-additive and non-white process and the reduction of speckle without blurring sharp features is known to be difficult. The new speckle reduction algorithm presented in this article utilizes a nonhomogeneous filter that adapts to the proximity and direction of the nearest important features. To remove speckle without blurring important features, the location and direction of edges in the image are estimated. Then for each pixel in the image, the distance and angle to the nearest edge are efficiently computed by a two-pass algorithm and stored in distance and angle maps. Finally for each pixel, an adaptive directional filter aligned to the nearest edge is applied. The shape and orientation of the adaptive filter are determined from the distance and angle maps. The new speckle reduction algorithm is tested with both synthesized and real ultrasound images. The performance of the new algorithm is also compared with those of other speckle reduction approaches and it is shown that the new algorithm performs favorably in reducing speckle without blurring important features.     

SCS-Net: A Scale and Context Sensitive Network for Retinal Vessel Segmentation

Accurately segmenting retinal vessel from retinal images is essential for the detection and diagnosis of many eye diseases. However, it remains a challenging task due to (1) the large variations of scale in the retinal vessels and (2) the complicated anatomical context of retinal vessels, including complex vasculature and morphology, the low contrast between some vessels and the background, and the existence of exudates and hemorrhage. It is difficult for a model to capture representative and distinguishing features for retinal vessels under such large scale and semantics variations. Limited training data also make this task even harder. In order to comprehensively tackle these challenges, we propose a novel scale and context sensitive network (a.k.a., SCS⁻Net) for retinal vessel segmentation. We first propose a scale-aware feature aggregation (SFA) module, aiming at dynamically adjusting the receptive fields to effectively extract multi-scale features. Then, an adaptive feature fusion (AFF) module is designed to guide efficient fusion between adjacent hierarchical features to capture more semantic information. Finally, a multi-level semantic supervision (MSS) module is employed to learn more distinctive semantic representation for refining the vessel maps. We conduct extensive experiments on the six mainstream retinal image databases (DRIVE, CHASEDB1, STARE, IOSTAR, HRF, and LES-AV). The experimental results demonstrate the effectiveness of the proposed SCS-Net, which is capable of achieving better segmentation performance than other state-of-the-art approaches, especially for the challenging cases with large scale variations and complex context environments.     

Feature-based non-parametric estimation of Kullback–Leibler divergence for SAR image change detection

In this article, a method based on a non-parametric estimation of the Kullback–Leibler divergence using a local feature space is proposed for synthetic aperture radar (SAR) image change detection. First, local features based on a set of Gabor filters are extracted from both pre- and post-event images. The distribution of these local features from a local neighbourhood is considered as a statistical representation of the local image information. The Kullback–Leibler divergence as a probabilistic distance is used for measuring the similarity of the two distributions. Nevertheless, it is not trivial to estimate the distribution of a high-dimensional random vector, let alone the comparison of two distributions. Thus, a non-parametric method based on k-nearest neighbour search is proposed to compute the Kullback–Leibler divergence between the two distributions. Through experiments, this method is compared with other state-of-the-art methods and the effectiveness of the proposed method for SAR image change detection is demonstrated.     

Spatial contextual classification of remote sensing images using a Gaussian process

Gaussian process (GP) classifiers represent a powerful and interesting theoretical framework for the Bayesian classification of remote sensing images. However, the integration of spatial information in GP classifier is still an open question, while researches have demonstrated that the classification results could be improved when the spatial information is used. In this context, in order to improve the performance of the traditional GP classifier, we propose to use Markov random fields (MRFs) to refine the classification results with the neighbourhood information in the images. In the proposed method (denoted as GP-MRF), the MRF model is used as a post-processing step to the pixelwise results with GP classifier which classifies each pixel in the image separately. Therefore, the proposed GP-MRF approach promotes solutions in which adjacent pixels are likely to belong to the same class. Experimental results show that the GP-MRF could achieve better classification accuracy compared to the original GP classifier and the state-of-the-art spatial contextual classification methods.     

Weakly supervised histopathology image segmentation with self-attention

Accurate segmentation in histopathology images at pixel-level plays a critical role in the digital pathology workflow. The development of weakly supervised methods for histopathology image segmentation liberates pathologists from time-consuming and labor-intensive works, opening up possibilities of further automated quantitative analysis of whole-slide histopathology images. As an effective subgroup of weakly supervised methods, multiple instance learning (MIL) has achieved great success in histopathology images. In this paper, we specially treat pixels as instances so that the histopathology image segmentation task is transformed into an instance prediction task in MIL. However, the lack of relations between instances in MIL limits the further improvement of segmentation performance. Therefore, we propose a novel weakly supervised method called SA-MIL for pixel-level segmentation in histopathology images. SA-MIL introduces a self-attention mechanism into the MIL framework, which captures global correlation among all instances. In addition, we use deep supervision to make the best use of information from limited annotations in the weakly supervised method. Our approach makes up for the shortcoming that instances are independent of each other in MIL by aggregating global contextual information. We demonstrate state-of-the-art results compared to other weakly supervised methods on two histopathology image datasets. It is evident that our approach has generalization ability for the high performance on both tissue and cell histopathology datasets. There is potential in our approach for various applications in medical images.     

Real‐time Interpolation for True 3‐Dimensional Ultrasound Image Volumes

We compared trilinear interpolation to voxel nearest neighbor and distance‐weighted algorithms for fast and accurate processing of true 3‐dimensional ultrasound (3DUS) image volumes. In this study, the computational efficiency and interpolation accuracy of the 3 methods were compared on the basis of a simulated 3DUS image volume, 34 clinical 3DUS image volumes from 5 patients, and 2 experimental phantom image volumes. We show that trilinear interpolation improves interpolation accuracy over both the voxel nearest neighbor and distance‐weighted algorithms yet achieves real‐time computational performance that is comparable to the voxel nearest neighbor algrorithm (1–2 orders of magnitude faster than the distance‐weighted algorithm) as well as the fastest pixel‐based algorithms for processing tracked 2‐dimensional ultrasound images (0.035 seconds per 2‐dimesional cross‐sectional image [76,800 pixels interpolated, or 0.46 ms/1000 pixels] and 1.05 seconds per full volume with a 1‐mm³ voxel size [4.6 million voxels interpolated, or 0.23 ms/1000 voxels]). On the basis of these results, trilinear interpolation is recommended as a fast and accurate interpolation method for rectilinear sampling of 3DUS image acquisitions, which is required to facilitate subsequent processing and display during operating room procedures such as image‐guided neurosurgery.