Semi-supervised training of deep convolutional neural networks with heterogeneous data and few local annotations: An experiment on prostate histopathology image classification

Convolutional neural networks (CNNs) are state-of-the-art computer vision techniques for various tasks, particularly for image classification. However, there are domains where the training of classification models that generalize on several datasets is still an open challenge because of the highly heterogeneous data and the lack of large datasets with local annotations of the regions of interest, such as histopathology image analysis. Histopathology concerns the microscopic analysis of tissue specimens processed in glass slides to identify diseases such as cancer. Digital pathology concerns the acquisition, management and automatic analysis of digitized histopathology images that are large, having in the order of ${100}^{\prime }{000}^2$ pixels per image. Digital histopathology images are highly heterogeneous due to the variability of the image acquisition procedures. Creating locally labeled regions (required for the training) is time-consuming and often expensive in the medical field, as physicians usually have to annotate the data. Despite the advances in deep learning, leveraging strongly and weakly annotated datasets to train classification models is still an unsolved problem, mainly when data are very heterogeneous. Large amounts of data are needed to create models that generalize well. This paper presents a novel approach to train CNNs that generalize to heterogeneous datasets originating from various sources and without local annotations. The data analysis pipeline targets Gleason grading on prostate images and includes two models in sequence, following a teacher/student training paradigm. The teacher model (a high-capacity neural network) automatically annotates a set of pseudo-labeled patches used to train the student model (a smaller network). The two models are trained with two different teacher/student approaches: semi-supervised learning and semi-weekly supervised learning. For each of the two approaches, three student training variants are presented. The baseline is provided by training the student model only with the strongly annotated data. Classification performance is evaluated on the student model at the patch level (using the local annotations of the Tissue Micro-Arrays Zurich dataset) and at the global level (using the TCGA-PRAD, The Cancer Genome Atlas-PRostate ADenocarcinoma, whole slide image Gleason score). The teacher/student paradigm allows the models to better generalize on both datasets, despite the inter-dataset heterogeneity and the small number of local annotations used. The classification performance is improved both at the patch-level (up to $\kappa =0.6127\pm 0.0133$ from $\kappa =0.5667\pm 0.0285$), at the TMA core-level (Gleason score) (up to $\kappa =0.7645\pm 0.0231$ from $\kappa =0.7186\pm 0.0306$) and at the WSI-level (Gleason score) (up to $\kappa =0.4529\pm 0.0512$ from $\kappa =0.2293\pm 0.1350$). The results show that with the teacher/student paradigm, it is possible to train models that generalize on datasets from entirely different sources, despite the inter-dataset heterogeneity and the lack of large datasets with local annotations.     

Detection,segmentation, and 3D pose estimation of surgical tools using convolutional neural networks and algebraic geometry

Background and objective:Surgical tool detection, segmentation, and 3D pose estimation are crucial components in Computer-Assisted Laparoscopy (CAL). The existing frameworks have two main limitations. First, they do not integrate all three components. Integration is critical; for instance, one should not attempt computing pose if detection is negative. Second, they have highly specific requirements, such as the availability of a CAD model. We propose an integrated and generic framework whose sole requirement for the 3D pose is that the tool shaft is cylindrical. Our framework makes the most of deep learning and geometric 3D vision by combining a proposed Convolutional Neural Network (CNN) with algebraic geometry. We show two applications of our framework in CAL: tool-aware rendering in Augmented Reality (AR) and tool-based 3D measurement. Methods:We name our CNN as ART-Net (Augmented Reality Tool Network). It has a Single Input Multiple Output (SIMO) architecture with one encoder and multiple decoders to achieve detection, segmentation, and geometric primitive extraction. These primitives are the tool edge-lines, mid-line, and tip. They allow the tool’s 3D pose to be estimated by a fast algebraic procedure. The framework only proceeds if a tool is detected. The accuracy of segmentation and geometric primitive extraction is boosted by a new Full resolution feature map Generator (FrG). We extensively evaluate the proposed framework with the EndoVis and new proposed datasets. We compare the segmentation results against several variants of the Fully Convolutional Network (FCN) and U-Net. Several ablation studies are provided for detection, segmentation, and geometric primitive extraction. The proposed datasets are surgery videos of different patients.Results:In detection, ART-Net achieves $100.0\%$ in both average precision and accuracy. In segmentation, it achieves $81.0\%$ in mean Intersection over Union (mIoU) on the robotic EndoVis dataset (articulated tool), where it outperforms both FCN and U-Net, by $4.5pp$ and $2.9pp,$ respectively. It achieves $88.2\%$ in mIoU on the remaining datasets (non-articulated tool). In geometric primitive extraction, ART-Net achieves $2.45{}^{\circ }$ and $2.23{}^{\circ }$ in mean Arc Length (mAL) error for the edge-lines and mid-line, respectively, and 9.3 pixels in mean Euclidean distance error for the tool-tip. Finally, in terms of 3D pose evaluated on animal data, our framework achieves 1.87 mm, 0.70 mm, and 4.80 mm mean absolute errors on the $X,$ $Y,$ and $Z$ coordinates, respectively, and $5.{94}^{\circ }$ angular error on the shaft orientation. It achieves 2.59 mm and 1.99 mm in mean and median location error of the tool head evaluated on patient data.Conclusions:The proposed framework outperforms existing ones in detection and segmentation. Compared to separate networks, integrating the tasks in a single network preserves accuracy in detection and segmentation but substantially improves accuracy in geometric primitive extraction. Overall, our framework has similar or better accuracy in 3D pose estimation while largely improving robustness against the very challenging imaging conditions of laparoscopy. The source code of our framework and our annotated dataset will be made publicly available at https://github.com/kamruleee51/ART-Net.     

Unsupervised PET logan parametric image estimation using conditional deep image prior

Recently, deep learning-based denoising methods have been gradually used for PET images denoising and have shown great achievements. Among these methods, one interesting framework is conditional deep image prior (CDIP) which is an unsupervised method that does not need prior training or a large number of training pairs. In this work, we combined CDIP with Logan parametric image estimation to generate high-quality parametric images. In our method, the kinetic model is the Logan reference tissue model that can avoid arterial sampling. The neural network was utilized to represent the images of Logan slope and intercept. The patient’s computed tomography (CT) image or magnetic resonance (MR) image was used as the network input to provide anatomical information. The optimization function was constructed and solved by the alternating direction method of multipliers (ADMM) algorithm. Both simulation and clinical patient datasets demonstrated that the proposed method could generate parametric images with more detailed structures. Quantification results showed that the proposed method results had higher contrast-to-noise (CNR) improvement ratios (PET/CT datasets: $62.25\%\pm 29.93\%$; striatum of brain PET datasets : $129.51\%\pm 32.13\%$, thalamus of brain PET datasets: $128.24\%\pm 31.18\%$) than Gaussian filtered results (PET/CT datasets: $23.33\%\pm 18.63\%$; striatum of brain PET datasets: $74.71\%\pm 8.71\%$, thalamus of brain PET datasets: $73.02\%\pm 9.34\%$) and nonlocal mean (NLM) denoised results (PET/CT datasets: $37.55\%\pm 26.56\%$; striatum of brain PET datasets: $100.89\%\pm 16.13\%$, thalamus of brain PET datasets: $103.59\%\pm 16.37\%$).     

Investigating the Accumulation of Submicron Phase-Change Droplets in Tumors

Submicron phase-change droplets are an emerging class of ultrasound contrast agent. Compared with microbubbles, their relatively small size and increased stability offer the potential to passively extravasate and accumulate in solid tumors through the enhanced permeability and retention effect. Under exposure to sufficiently powerful ultrasound, these droplets can convert into in situ gas microbubbles and thus be used as an extravascular-specific contrast agent. However, in vivo imaging methods to detect extravasated droplets have yet to be established. Here, we develop an ultrasound imaging pulse sequence within diagnostic safety limits to selectively detect droplet extravasation in tumors. Tumor-bearing mice were injected with submicron perfluorobutane droplets and interrogated with our imaging–vaporization–imaging sequence. By use of a pulse subtraction method, median droplet extravasation signal relative to the total signal within the tumor was estimated to be $E_{\text{tumor}}=37\pm 5\%$ compared with the kidney $E_{\text{kidney}}=-2\pm 8\%$ (p < 0.001). This work contributes toward the advancement of volatile phase-shift droplets as a next-generation ultrasound agent for imaging and therapy.     

Residual cyclegan for robust domain transformation of histopathological tissue slides

Variation between stains in histopathology is commonplace across different medical centers. This can have a significant effect on the reliability of machine learning algorithms. In this paper, we propose to reduce performance variability by using -consistent generative adversarial (CycleGAN) networks to remove staining variation. We improve upon the regular CycleGAN by incorporating residual learning. We comprehensively evaluate the performance of our stain transformation method and compare its usefulness in addition to extensive data augmentation to enhance the robustness of tissue segmentation algorithms. Our steps are as follows: first, we train a model to perform segmentation on tissue slides from a single source center, while heavily applying augmentations to increase robustness to unseen data. Second, we evaluate and compare the segmentation performance on data from other centers, both with and without applying our CycleGAN stain transformation. We compare segmentation performances in a colon tissue segmentation and kidney tissue segmentation task, covering data from 6 different centers. We show that our transformation method improves the overall Dice coefficient by $9\%$ over the non-normalized target data and by $4\%$ over traditional stain transformation in our colon tissue segmentation task. For kidney segmentation, our residual CycleGAN increases performance by $10\%$ over no transformation and around $2\%$ compared to the non-residual CycleGAN.