Automated left ventricular segmentation from cardiac magnetic resonance images via adversarial learning with multi-stage pose estimation network and co-discriminator

Left ventricular (LV) segmentation is essential for the early diagnosis of cardiovascular diseases, which has been reported as the leading cause of death all over the world. However, automated LV segmentation from cardiac magnetic resonance images (CMRI) using the traditional convolutional neural networks (CNNs) is still a challenging task due to the limited labeled CMRI data and low tolerances to irregular scales, shapes and deformations of LV. In this paper, we propose an automated LV segmentation method based on adversarial learning by integrating a multi-stage pose estimation network (MSPN) and a co-discrimination network. Different from existing CNNs, we use a MSPN with multi-scale dilated convolution (MDC) modules to enhance the ranges of receptive field for deep feature extraction. To fully utilize both labeled and unlabeled CMRI data, we propose a novel generative adversarial network (GAN) framework for LV segmentation by combining MSPN with co-discrimination networks. Specifically, the labeled CMRI are first used to initialize our segmentation network (MSPN) and co-discrimination network. Our GAN training includes two different kinds of epochs fed with both labeled and unlabeled CMRI data alternatively, which are different from the traditional CNNs only relied on the limited labeled samples to train the segmentation networks. As both ground truth and unlabeled samples are involved in guiding training, our method not only can converge faster but also obtain a better performance in LV segmentation. Our method is evaluated using MICCAI 2009 and 2017 challenge databases. Experimental results show that our method has obtained promising performance in LV segmentation, which also outperforms the state-of-the-art methods in terms of LV segmentation accuracy from the comparison results.     

Esophageal optical coherence tomography image synthesis using an adversarially learned variational autoencoder

Endoscopic optical coherence tomography (OCT) imaging offers a non-invasive way to detect esophageal lesions on the microscopic scale, which is of clinical potential in the early diagnosis and treatment of esophageal cancers. Recent studies focused on applying deep learning-based methods in esophageal OCT image analysis and achieved promising results, which require a large data size. However, traditional data augmentation techniques generate samples that are highly correlated and sometimes far from reality, which may not lead to a satisfied trained model. In this paper, we proposed an adversarial learned variational autoencoder (AL-VAE) to generate high-quality esophageal OCT samples. The AL-VAE combines the generative adversarial network (GAN) and variational autoencoder (VAE) in a simple yet effective way, which preserves the advantages of VAEs, such as stable training and nice latent manifold, and requires no extra discriminators. Experimental results verified the proposed method achieved better image quality in generating esophageal OCT images when compared with the state-of-the-art image synthesis network, and its potential in improving deep learning model performance was also evaluated by esophagus segmentation.     

Boundary-aware context neural network for medical image segmentation

Medical image segmentation can provide a reliable basis for further clinical analysis and disease diagnosis. With the development of convolutional neural networks (CNNs), medical image segmentation performance has advanced significantly. However, most existing CNN-based methods often produce unsatisfactory segmentation masks without accurate object boundaries. This problem is caused by the limited context information and inadequate discriminative feature maps after consecutive pooling and convolution operations. Additionally, medical images are characterized by high intra-class variation, inter-class indistinction and noise, extracting powerful context and aggregating discriminative features for fine-grained segmentation remain challenging. In this study, we formulate a boundary-aware context neural network (BA-Net) for 2D medical image segmentation to capture richer context and preserve fine spatial information, which incorporates encoder-decoder architecture. In each stage of the encoder sub-network, a proposed pyramid edge extraction module first obtains multi-granularity edge information. Then a newly designed mini multi-task learning module for jointly learning segments the object masks and detects lesion boundaries, in which a new interactive attention layer is introduced to bridge the two tasks. In this way, information complementarity between different tasks is achieved, which effectively leverages the boundary information to offer strong cues for better segmentation prediction. Finally, a cross feature fusion module acts to selectively aggregate multi-level features from the entire encoder sub-network. By cascading these three modules, richer context and fine-grain features of each stage are encoded and then delivered to the decoder. The results of extensive experiments on five datasets show that the proposed BA-Net outperforms state-of-the-art techniques.     

Attention convolutional neural network for accurate segmentation and quantification of lesions in ischemic stroke disease

Ischemic stroke lesion and white matter hyperintensity (WMH) lesion appear as regions of abnormally signal intensity on magnetic resonance image (MRI) sequences. Ischemic stroke is a frequent cause of death and disability, while WMH is a risk factor for stroke. Accurate segmentation and quantification of ischemic stroke and WMH lesions are important for diagnosis and prognosis. However, radiologists have a difficult time distinguishing these two types of similar lesions. A novel deep residual attention convolutional neural network (DRANet) is proposed to accurately and simultaneously segment and quantify ischemic stroke and WMH lesions in the MRI images. DRANet inherits the advantages of the U-net design and applies a novel attention module that extracts high-quality features from the input images. Moreover, the Dice loss function is used to train DRANet to address data imbalance in the training data set. DRANet is trained and evaluated on 742 2D MRI images which are produced from the sub-acute ischemic stroke lesion segmentation (SISS) challenge. Empirical tests demonstrate that DRANet outperforms several other state-of-the-art segmentation methods. It accurately segments and quantifies both ischemic stroke lesion and WMH. Ablation experiments reveal that attention modules improve the predictive performance of DRANet.     

Image-to-image translation of label-free molecular vibrational images for a histopathological review using the UNet+/seg-cGAN model

Translating images generated by label-free microscopy imaging, such as Coherent Anti-Stokes Raman Scattering (CARS), into more familiar clinical presentations of histopathological images will help the adoption of real-time, spectrally resolved label-free imaging in clinical diagnosis. Generative adversarial networks (GAN) have made great progress in image generation and translation, but have been criticized for lacking precision. In particular, GAN has often misinterpreted image information and identified incorrect content categories during image translation of microscopy scans. To alleviate this problem, we developed a new Pix2pix GAN model that simultaneously learns classifying contents in the images from a segmentation dataset during the image translation training. Our model integrates UNet+ with seg-cGAN, conditional generative adversarial networks with partial regularization of segmentation. Technical innovations of the UNet+/seg-cGAN model include: (1) replacing UNet with UNet+ as the Pix2pix cGAN’s generator to enhance pattern extraction and richness of the gradient, and (2) applying the partial regularization strategy to train a part of the generator network as the segmentation sub-model on a separate segmentation dataset, thus enabling the model to identify correct content categories during image translation. The quality of histopathological-like images generated based on label-free CARS images has been improved significantly.     

Rethinking adversarial domain adaptation: Orthogonal decomposition for unsupervised domain adaptation in medical image segmentation

Medical image segmentation methods based on deep learning have made remarkable progress. However, such existing methods are sensitive to data distribution. Therefore, slight domain shifts will cause a decline of performance in practical applications. To relieve this problem, many domain adaptation methods learn domain-invariant representations by alignment or adversarial training whereas ignoring domain-specific representations. In response to this issue, this paper rethinks the traditional domain adaptation framework and proposes a novel orthogonal decomposition adversarial domain adaptation (ODADA) architecture for medical image segmentation. The main idea behind our proposed ODADA model is to decompose the input features into domain-invariant and domain-specific representations and then use the newly designed orthogonal loss function to encourage their independence. Furthermore, we propose a two-step optimization strategy to extract domain-invariant representations by separating domain-specific representations, fighting the performance degradation caused by domain shifts. Encouragingly, the proposed ODADA framework is plug-and-play and can replace the traditional adversarial domain adaptation module. The proposed method has consistently demonstrated effectiveness through comprehensive experiments on three publicly available datasets, including cross-site prostate segmentation dataset, cross-site COVID-19 lesion segmentation dataset, and cross-modality cardiac segmentation dataset. The source code is available at https://github.com/YonghengSun1997/ODADA.     

Automated cardiac segmentation of cross-modal medical images using unsupervised multi-domain adaptation and spatial neural attention structure

Accurate cardiac segmentation of multimodal images, e.g., magnetic resonance (MR), computed tomography (CT) images, plays a pivot role in auxiliary diagnoses, treatments and postoperative assessments of cardiovascular diseases. However, training a well-behaved segmentation model for the cross-modal cardiac image analysis is challenging, due to their diverse appearances/distributions from different devices and acquisition conditions. For instance, a well-trained segmentation model based on the source domain of MR images is often failed in the segmentation of CT images. In this work, a cross-modal images-oriented cardiac segmentation scheme is proposed using a symmetric full convolutional neural network (SFCNN) with the unsupervised multi-domain adaptation (UMDA) and a spatial neural attention (SNA) structure, termed UMDA-SNA-SFCNN, having the merits of without the requirement of any annotation on the test domain. Specifically, UMDA-SNA-SFCNN incorporates SNA to the classic adversarial domain adaptation network to highlight the relevant regions, while restraining the irrelevant areas in the cross-modal images, so as to suppress the negative transfer in the process of unsupervised domain adaptation. In addition, the multi-layer feature discriminators and a predictive segmentation-mask discriminator are established to connect the multi-layer features and segmentation mask of the backbone network, SFCNN, to realize the fine-grained alignment of unsupervised cross-modal feature domains. Extensive confirmative and comparative experiments on the benchmark Multi-Modality Whole Heart Challenge dataset show that the proposed model is superior to the state-of-the-art cross-modal segmentation methods.     

Multi-constraint generative adversarial network for dose prediction in radiotherapy

Radiation therapy (RT) is regarded as the primary treatment for cancer in the clinic, aiming to deliver an accurate dose to the planning target volume (PTV) while protecting the surrounding organs at risk (OARs). To improve the effectiveness of the treatment planning, deep learning methods are widely adopted to predict dose distribution maps for clinical treatment planning. In this paper, we present a novel multi-constraint dose prediction model based on generative adversarial network, named Mc-GAN, to automatically predict the dose distribution map from the computer tomography (CT) images and the masks of PTV and OARs. Specifically, the generator is an embedded UNet-like structure with dilated convolution to capture both the global and local information. During the feature extraction, a dual attention module (DAM) is embedded to force the generator to take more heed of internal semantic relevance. To improve the prediction accuracy, two additional losses, i.e., the locality-constrained loss (LCL) and the self-supervised perceptual loss (SPL), are introduced besides the conventional global pixel-level loss and adversarial loss. Concretely, the LCL tries to focus on the predictions of locally important areas while the SPL aims to prevent the predicted dose maps from the possible distortion at the feature level. Evaluated on two in-house datasets, our proposed Mc-GAN has been demonstrated to outperform other state-of-the-art methods in almost all PTV and OARs criteria.     

Sketch guided and progressive growing GAN for realistic and editable ultrasound image synthesis

Ultrasound (US) imaging is widely used for anatomical structure inspection in clinical diagnosis. The training of new sonographers and deep learning based algorithms for US image analysis usually requires a large amount of data. However, obtaining and labeling large-scale US imaging data are not easy tasks, especially for diseases with low incidence. Realistic US image synthesis can alleviate this problem to a great extent. In this paper, we propose a generative adversarial network (GAN) based image synthesis framework. Our main contributions include: (1) we present the first work that can synthesize realistic B-mode US images with high-resolution and customized texture editing features; (2) to enhance structural details of generated images, we propose to introduce auxiliary sketch guidance into a conditional GAN. We superpose the edge sketch onto the object mask and use the composite mask as the network input; (3) to generate high-resolution US images, we adopt a progressive training strategy to gradually generate high-resolution images from low-resolution images. In addition, a feature loss is proposed to minimize the difference of high-level features between the generated and real images, which further improves the quality of generated images; (4) the proposed US image synthesis method is quite universal and can also be generalized to the US images of other anatomical structures besides the three ones tested in our study (lung, hip joint, and ovary); (5) extensive experiments on three large US image datasets are conducted to validate our method. Ablation studies, customized texture editing, user studies, and segmentation tests demonstrate promising results of our method in synthesizing realistic US images.     

Learning to complete incomplete hearts for population analysis of cardiac MR images

Cardiac MR acquisition with complete coverage from base to apex is required to ensure accurate subsequent analyses, such as volumetric and functional measurements. However, this requirement cannot be guaranteed when acquiring images in the presence of motion induced by cardiac muscle contraction and respiration. To address this problem, we propose an effective two-stage pipeline for detecting and synthesising absent slices in both the apical and basal region. The detection model comprises several dense blocks containing convolutional long short-term memory (ConvLSTM) layers, to leverage through-plane contextual and sequential ordering information of slices in cine MR data and achieve reliable classification results. The imputation network is based on a dedicated conditional generative adversarial network (GAN) that helps retain key visual cues and fine structural details in the synthesised image slices. The proposed network can infer multiple missing slices that are anatomically plausible and lead to improved accuracy of subsequent analyses on cardiac MRIs, e.g., ventricle segmentation, cardiac quantification compared to those derived from incomplete cardiac MR datasets. For instance, the results obtained when compensating for the absence of two basal slices show that the mean differences to the reference of stroke volume and ejection fraction are only -1.3 mL and -1.0%, respectively, which are significantly smaller than those calculated from the incomplete data (-26.8 mL and -6.7%). The proposed approach can improve the reliability of high-throughput image analysis in large-scale population studies, minimising the need for re-scanning patients or discarding incomplete acquisitions.     

Segmentation and quantification of infarction without contrast agents via spatiotemporal generative adversarial learning

Accurate and simultaneous segmentation and full quantification (all indices are required in a clinical assessment) of the myocardial infarction (MI) area are crucial for early diagnosis and surgical planning. Current clinical methods remain subject to potential high-risk, nonreproducibility and time-consumption issues. In this study, a deep spatiotemporal adversarial network (DSTGAN) is proposed as a contrast-free, stable and automatic clinical tool to simultaneously segment and quantify MIs directly from the cine MR image. The DSTGAN is implemented using a conditional generative model, which conditions the distributions of the objective cine MR image to directly optimize the generalized error of the mapping between the input and the output. The method consists of the following: (1) A multi-level and multi-scale spatiotemporal variation encoder learns a coarse to fine hierarchical feature to effectively encode the MI-specific morphological and kinematic abnormality structures, which vary for different spatial locations and time periods. (2) The top-down and cross-task generators learn the shared representations between segmentation and quantification to use the commonalities and differences between the two related tasks and enhance the generator preference. (3) Three inter-/intra-tasks to label the relatedness discriminators are iteratively imposed on the encoder and generator to detect and correct the inconsistencies in the label relatedness between and within tasks via adversarial learning. Our proposed method yields a pixel classification accuracy of 96.98%, and the mean absolute error of the MI centroid is 0.96 mm from 165 clinical subjects. These results indicate the potential of our proposed method in aiding standardized MI assessments.     

Contrast agent-free synthesis and segmentation of ischemic heart disease images using progressive sequential causal GANs

The elimination of gadolinium contrast agent (CA) injections and manual segmentation are crucial for ischemic heart disease (IHD) diagnosis and treatment. In the clinic, CA-based late gadolinium enhancement (LGE) imaging and manual segmentation remain subject to concerns about potential toxicity, interobserver variability, and ineffectiveness. In this study, progressive sequential causal GANs (PSCGAN) are proposed. This is the first one-stop CA-free IHD technology that can simultaneously synthesize an LGE-equivalent image and segment diagnosis-related tissues (i.e., scars, healthy myocardium, blood pools, and other pixels) from cine MR images. To this end, the PSCGAN offer three unique properties: 1) a progressive framework that cascades three phases (i.e., priori generation, conditional synthesis, and enhanced segmentation) for divide-and-conquer training synthesis and segmentation of images. Importantly, this framework leverages the output of the previous phase as a priori condition to input the next phase and guides its training for enhancing performance, 2) a sequential causal learning network (SCLN) that creates a multi-scale, two-stream pathway and a multi-attention weighing unit to extract spatial and temporal dependencies from cine MR images and effectively select task-specific dependence. It also integrates the GAN architecture to leverage adversarial training to further facilitate the learning of interest dependencies of the latent space of cine MR images in all phases; and 3) two specifically designed self-learning loss terms: a synthetic regularization loss term leverages the spare regularization to avoid noise during synthesis, and a segmentation auxiliary loss term leverages the number of pixels for each tissue to compensate for discrimination during segmentation. Thus, the PSCGAN gain unprecedented performance while stably training in both synthesis and segmentation. By training and testing a total of 280 clinical subjects, our PSCGAN yield a synthetic normalization root-mean-squared-error of 0.14 and an overall segmentation accuracy of 97.17%. It also produces a 0.96 correlation coefficient for the scar ratio in a real diagnostic metric evaluation. These results proved that our method is able to offer significant assistance in the standardized assessment of cardiac disease.     

Ultrasound deep beamforming using a multiconstrained hybrid generative adversarial network

Ultrasound beamforming is a principal factor in high-quality ultrasound imaging. The conventional delay-and-sum (DAS) beamformer generates images with high computational speed but low spatial resolution; thus, many adaptive beamforming methods have been introduced to improve image qualities. However, these adaptive beamforming methods suffer from high computational complexity, which limits their practical applications. Hence, an advanced beamformer that can overcome spatiotemporal resolution bottlenecks is eagerly awaited. In this paper, we propose a novel deep-learning-based algorithm, called the multiconstrained hybrid generative adversarial network (MC-HGAN) beamformer that rapidly achieves high-quality ultrasound imaging. The MC-HGAN beamformer directly establishes a one-shot mapping between the radio frequency signals and the reconstructed ultrasound images through a hybrid generative adversarial network (GAN) model. Through two specific branches, the hybrid GAN model extracts both radio frequency-based and image-based features and integrates them through a fusion module. We also introduce a multiconstrained training strategy to provide comprehensive guidance for the network by invoking intermediates to co-constrain the training process. Moreover, our beamformer is designed to adapt to various ultrasonic emission modes, which improves its generalizability for clinical applications. We conducted experiments on a variety of datasets scanned by line-scan and plane wave emission modes and evaluated the results with both similarity-based and ultrasound-specific metrics. The comparisons demonstrate that the MC-HGAN beamformer generates ultrasound images whose quality is higher than that of images generated by other deep learning-based methods and shows very high robustness in different clinical datasets. This technology also shows great potential in real-time imaging.     

Enhancing MR image segmentation with realistic adversarial data augmentation

The success of neural networks on medical image segmentation tasks typically relies on large labeled datasets for model training. However, acquiring and manually labeling a large medical image set is resource-intensive, expensive, and sometimes impractical due to data sharing and privacy issues. To address this challenge, we propose AdvChain, a generic adversarial data augmentation framework, aiming at improving both the diversity and effectiveness of training data for medical image segmentation tasks. AdvChain augments data with dynamic data augmentation, generating randomly chained photo-metric and geometric transformations to resemble realistic yet challenging imaging variations to expand training data. By jointly optimizing the data augmentation model and a segmentation network during training, challenging examples are generated to enhance network generalizability for the downstream task. The proposed adversarial data augmentation does not rely on generative networks and can be used as a plug-in module in general segmentation networks. It is computationally efficient and applicable for both low-shot supervised and semi-supervised learning. We analyze and evaluate the method on two MR image segmentation tasks: cardiac segmentation and prostate segmentation with limited labeled data. Results show that the proposed approach can alleviate the need for labeled data while improving model generalization ability, indicating its practical value in medical imaging applications.     

An end-to-end network for segmenting the vasculature of three retinal capillary plexuses from OCT angiographic volumes

The segmentation of en face retinal capillary angiograms from volumetric optical coherence tomographic angiography (OCTA) usually relies on retinal layer segmentation, which is time-consuming and error-prone. In this study, we developed a deep-learning-based method to segment vessels in the superficial vascular plexus (SVP), intermediate capillary plexus (ICP), and deep capillary plexus (DCP) directly from volumetric OCTA data. The method contains a three-dimensional convolutional neural network (CNN) for extracting distinct retinal layers, a custom projection module to generate three vascular plexuses from OCTA data, and three parallel CNNs to segment vasculature. Experimental results on OCTA data from rat eyes demonstrated the feasibility of the proposed method. This end-to-end network has the potential to simplify OCTA data processing on retinal vasculature segmentation. The main contribution of this study is that we propose a custom projection module to connect retinal layer segmentation and vasculature segmentation modules and automatically convert data from three to two dimensions, thus establishing an end-to-end method to segment three retinal capillary plexuses from volumetric OCTA without any human intervention.     

GDCSeg-Net: general optic disc and cup segmentation network for multi-device fundus images

Accurate segmentation of optic disc (OD) and optic cup (OC) in fundus images is crucial for the analysis of many retinal diseases, such as the screening and diagnosis of glaucoma and atrophy segmentation. Due to domain shift between different datasets caused by different acquisition devices and modes and inadequate training caused by small sample dataset, the existing deep-learning-based OD and OC segmentation networks have poor generalization ability for different fundus image datasets. In this paper, adopting the mixed training strategy based on different datasets for the first time, we propose an encoder-decoder based general OD and OC segmentation network (named as GDCSeg-Net) with the newly designed multi-scale weight-shared attention (MSA) module and densely connected depthwise separable convolution (DSC) module, to effectively overcome these two problems. Experimental results show that our proposed GDCSeg-Net is competitive with other state-of-the-art methods on five different public fundus image datasets, including REFUGE, MESSIDOR, RIM-ONE-R3, Drishti-GS and IDRiD.     

Dermoscopic image retrieval based on rotation-invariance deep hashing

Dermoscopic image retrieval technology can provide dermatologists with valuable information such as similar confirmed skin disease cases and diagnosis reports to assist doctors in their diagnosis. In this study, we design a dermoscopic image retrieval algorithm using convolutional neural networks (CNNs) and hash coding. A hybrid dilated convolution spatial attention module is proposed, which can focus on important information and suppress irrelevant information based on the complex morphological characteristics of dermoscopic images. Furthermore, we also propose a Cauchy rotation invariance loss function in view of the skin lesion target without the main direction. This function constrains CNNs to learn output differences in samples from different angles and to make CNNs obtain a certain rotation invariance. Extensive experiments are conducted on dermoscopic image datasets to verify the effectiveness and versatility of the proposed module, algorithm, and loss function. Experiment results show that the rotation-invariance deep hashing network with the proposed spatial attention module obtains better performance on the task of dermoscopic image retrieval.     

Dense regression activation maps for lesion segmentation in CT scans of COVID-19 patients

Automatic lesion segmentation on thoracic CT enables rapid quantitative analysis of lung involvement in COVID-19 infections. However, obtaining a large amount of voxel-level annotations for training segmentation networks is prohibitively expensive. Therefore, we propose a weakly-supervised segmentation method based on dense regression activation maps (dRAMs). Most weakly-supervised segmentation approaches exploit class activation maps (CAMs) to localize objects. However, because CAMs were trained for classification, they do not align precisely with the object segmentations. Instead, we produce high-resolution activation maps using dense features from a segmentation network that was trained to estimate a per-lobe lesion percentage. In this way, the network can exploit knowledge regarding the required lesion volume. In addition, we propose an attention neural network module to refine dRAMs, optimized together with the main regression task. We evaluated our algorithm on 90 subjects. Results show our method achieved 70.2% Dice coefficient, substantially outperforming the CAM-based baseline at 48.6%. We published our source code at https://github.com/DIAGNijmegen/bodyct-dram.     

Adversarial learning-based multi-level dense-transmission knowledge distillation for AP-ROP detection

The Aggressive Posterior Retinopathy of Prematurity (AP-ROP) is the major cause of blindness for premature infants. The automatic diagnosis method has become an important tool for detecting AP-ROP. However, most existing automatic diagnosis methods were with heavy complexity, which hinders the development of the detecting devices. Hence, a small network (student network) with a high imitation ability is exactly needed, which can mimic a large network (teacher network) with promising diagnostic performance. Also, if the student network is too small due to the increasing gap between teacher and student networks, the diagnostic performance will drop. To tackle the above issues, we propose a novel adversarial learning-based multi-level dense knowledge distillation method for detecting AP-ROP. Specifically, the pre-trained teacher network is utilized to train multiple intermediate-size networks (i.e., teacher-assistant networks) and one student network by dense transmission mode, where the knowledge from all upper-level networks is transmitted to the current lower-level network. To ensure that two adjacent networks can distill the abundant knowledge, the adversarial learning module is leveraged to enforce the lower-level network to generate the features that are similar to those of the upper-level network. Extensive experiments demonstrate that our proposed method can realize the effective knowledge distillation from the teacher to student networks. We achieve a promising knowledge distillation performance for our private dataset and a public dataset, which can provide a new insight for devising lightweight detecting systems of fundus diseases for practical use.