Super-resolution and segmentation deep learning for breast cancer histopathology image analysis

Traditionally, a high-performance microscope with a large numerical aperture is required to acquire high-resolution images. However, the images’ size is typically tremendous. Therefore, they are not conveniently managed and transferred across a computer network or stored in a limited computer storage system. As a result, image compression is commonly used to reduce image size resulting in poor image resolution. Here, we demonstrate custom convolution neural networks (CNNs) for both super-resolution image enhancement from low-resolution images and characterization of both cells and nuclei from hematoxylin and eosin (H&E) stained breast cancer histopathological images by using a combination of generator and discriminator networks so-called super-resolution generative adversarial network-based on aggregated residual transformation (SRGAN-ResNeXt) to facilitate cancer diagnosis in low resource settings. The results provide high enhancement in image quality where the peak signal-to-noise ratio and structural similarity of our network results are over 30 dB and 0.93, respectively. The derived performance is superior to the results obtained from both the bicubic interpolation and the well-known SRGAN deep-learning methods. In addition, another custom CNN is used to perform image segmentation from the generated high-resolution breast cancer images derived with our model with an average Intersection over Union of 0.869 and an average dice similarity coefficient of 0.893 for the H&E image segmentation results. Finally, we propose the jointly trained SRGAN-ResNeXt and Inception U-net Models, which applied the weights from the individually trained SRGAN-ResNeXt and inception U-net models as the pre-trained weights for transfer learning. The jointly trained model’s results are progressively improved and promising. We anticipate these custom CNNs can help resolve the inaccessibility of advanced microscopes or whole slide imaging (WSI) systems to acquire high-resolution images from low-performance microscopes located in remote-constraint settings.     

Deep learning solution for medical image localization and orientation detection

Magnetic Resonance (MR) imaging plays an important role in medical diagnosis and biomedical research. Due to the high in-slice resolution and low through-slice resolution nature of MR imaging, the usefulness of the reconstruction highly depends on the positioning of the slice group. Traditional clinical workflow relies on time-consuming manual adjustment that cannot be easily reproduced. Automation of this task can therefore bring important benefits in terms of accuracy, speed and reproducibility. Current auto-slice-positioning methods rely on automatically detected landmarks to derive the positioning, and previous studies suggest that a large, redundant set of landmarks are required to achieve robust results. However, a costly data curation procedure is needed to generate training labels for those landmarks, and the results can still be highly sensitive to landmark detection errors. More importantly, a set of anatomical landmark locations are not naturally produced during the standard clinical workflow, which makes online learning impossible. To address these limitations, we propose a novel framework for auto-slice-positioning that focuses on localizing the canonical planes within a 3D volume. The proposed framework consists of two major steps. A multi-resolution region proposal network is first used to extract a volume-of-interest, after which a V-net-like segmentation network is applied to segment the orientation planes. Importantly, our algorithm also includes a Performance Measurement Index as an indication of the algorithm’s confidence. We evaluate the proposed framework on both knee and shoulder MR scans. Our method outperforms state-of-the-art automatic positioning algorithms in terms of accuracy and robustness.     

Multi-modal contrastive mutual learning and pseudo-label re-learning for semi-supervised medical image segmentation

Semi-supervised learning has a great potential in medical image segmentation tasks with a few labeled data, but most of them only consider single-modal data. The excellent characteristics of multi-modal data can improve the performance of semi-supervised segmentation for each image modality. However, a shortcoming for most existing multi-modal solutions is that as the corresponding processing models of the multi-modal data are highly coupled, multi-modal data are required not only in the training but also in the inference stages, which thus limits its usage in clinical practice. Consequently, we propose a semi-supervised contrastive mutual learning (Semi-CML) segmentation framework, where a novel area-similarity contrastive (ASC) loss leverages the cross-modal information and prediction consistency between different modalities to conduct contrastive mutual learning. Although Semi-CML can improve the segmentation performance of both modalities simultaneously, there is a performance gap between two modalities, i.e., there exists a modality whose segmentation performance is usually better than that of the other. Therefore, we further develop a soft pseudo-label re-learning (PReL) scheme to remedy this gap. We conducted experiments on two public multi-modal datasets. The results show that Semi-CML with PReL greatly outperforms the state-of-the-art semi-supervised segmentation methods and achieves a similar (and sometimes even better) performance as fully supervised segmentation methods with 100% labeled data, while reducing the cost of data annotation by 90%. We also conducted ablation studies to evaluate the effectiveness of the ASC loss and the PReL module.     

A survey on active learning and human-in-the-loop deep learning for medical image analysis

Fully automatic deep learning has become the state-of-the-art technique for many tasks including image acquisition, analysis and interpretation, and for the extraction of clinically useful information for computer-aided detection, diagnosis, treatment planning, intervention and therapy. However, the unique challenges posed by medical image analysis suggest that retaining a human end-user in any deep learning enabled system will be beneficial. In this review we investigate the role that humans might play in the development and deployment of deep learning enabled diagnostic applications and focus on techniques that will retain a significant input from a human end user. Human-in-the-Loop computing is an area that we see as increasingly important in future research due to the safety-critical nature of working in the medical domain. We evaluate four key areas that we consider vital for deep learning in the clinical practice: (1) Active Learning to choose the best data to annotate for optimal model performance; (2) Interaction with model outputs - using iterative feedback to steer models to optima for a given prediction and offering meaningful ways to interpret and respond to predictions; (3) Practical considerations - developing full scale applications and the key considerations that need to be made before deployment; (4) Future Prospective and Unanswered Questions - knowledge gaps and related research fields that will benefit human-in-the-loop computing as they evolve. We offer our opinions on the most promising directions of research and how various aspects of each area might be unified towards common goals.     

Deep learning for detection and segmentation of artefact and disease instances in gastrointestinal endoscopy

The Endoscopy Computer Vision Challenge (EndoCV) is a crowd-sourcing initiative to address eminent problems in developing reliable computer aided detection and diagnosis endoscopy systems and suggest a pathway for clinical translation of technologies. Whilst endoscopy is a widely used diagnostic and treatment tool for hollow-organs, there are several core challenges often faced by endoscopists, mainly: 1) presence of multi-class artefacts that hinder their visual interpretation, and 2) difficulty in identifying subtle precancerous precursors and cancer abnormalities. Artefacts often affect the robustness of deep learning methods applied to the gastrointestinal tract organs as they can be confused with tissue of interest. EndoCV2020 challenges are designed to address research questions in these remits. In this paper, we present a summary of methods developed by the top 17 teams and provide an objective comparison of state-of-the-art methods and methods designed by the participants for two sub-challenges: i) artefact detection and segmentation (EAD2020), and ii) disease detection and segmentation (EDD2020). Multi-center, multi-organ, multi-class, and multi-modal clinical endoscopy datasets were compiled for both EAD2020 and EDD2020 sub-challenges. The out-of-sample generalization ability of detection algorithms was also evaluated. Whilst most teams focused on accuracy improvements, only a few methods hold credibility for clinical usability. The best performing teams provided solutions to tackle class imbalance, and variabilities in size, origin, modality and occurrences by exploring data augmentation, data fusion, and optimal class thresholding techniques.     

Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation

The medical imaging literature has witnessed remarkable progress in high-performing segmentation models based on convolutional neural networks. Despite the new performance highs, the recent advanced segmentation models still require large, representative, and high quality annotated datasets. However, rarely do we have a perfect training dataset, particularly in the field of medical imaging, where data and annotations are both expensive to acquire. Recently, a large body of research has studied the problem of medical image segmentation with imperfect datasets, tackling two major dataset limitations: scarce annotations where only limited annotated data is available for training, and weak annotations where the training data has only sparse annotations, noisy annotations, or image-level annotations. In this article, we provide a detailed review of the solutions above, summarizing both the technical novelties and empirical results. We further compare the benefits and requirements of the surveyed methodologies and provide our recommended solutions. We hope this survey article increases the community awareness of the techniques that are available to handle imperfect medical image segmentation datasets.     

MetricUNet: Synergistic image- and voxel-level learning for precise prostate segmentation via online sampling

Fully convolutional networks (FCNs), including UNet and VNet, are widely-used network architectures for semantic segmentation in recent studies. However, conventional FCN is typically trained by the cross-entropy or Dice loss, which only calculates the error between predictions and ground-truth labels for pixels individually. This often results in non-smooth neighborhoods in the predicted segmentation. This problem becomes more serious in CT prostate segmentation as CT images are usually of low tissue contrast. To address this problem, we propose a two-stage framework, with the first stage to quickly localize the prostate region, and the second stage to precisely segment the prostate by a multi-task UNet architecture. We introduce a novel online metric learning module through voxel-wise sampling in the multi-task network. Therefore, the proposed network has a dual-branch architecture that tackles two tasks: (1) a segmentation sub-network aiming to generate the prostate segmentation, and (2) a voxel-metric learning sub-network aiming to improve the quality of the learned feature space supervised by a metric loss. Specifically, the voxel-metric learning sub-network samples tuples (including triplets and pairs) in voxel-level through the intermediate feature maps. Unlike conventional deep metric learning methods that generate triplets or pairs in image-level before the training phase, our proposed voxel-wise tuples are sampled in an online manner and operated in an end-to-end fashion via multi-task learning. To evaluate the proposed method, we implement extensive experiments on a real CT image dataset consisting 339 patients. The ablation studies show that our method can effectively learn more representative voxel-level features compared with the conventional learning methods with cross-entropy or Dice loss. And the comparisons show that the proposed method outperforms the state-of-the-art methods by a reasonable margin.     

Deep learning toolbox for automated enhancement,segmentation, and graphing of cortical optical coherence tomography microangiograms

Optical coherence tomography angiography (OCTA) is becoming increasingly popular for neuroscientific study, but it remains challenging to objectively quantify angioarchitectural properties from 3D OCTA images. This is mainly due to projection artifacts or “tails” underneath vessels caused by multiple-scattering, as well as the relatively low signal-to-noise ratio compared to fluorescence-based imaging modalities. Here, we propose a set of deep learning approaches based on convolutional neural networks (CNNs) to automated enhancement, segmentation and gap-correction of OCTA images, especially of those obtained from the rodent cortex. Additionally, we present a strategy for skeletonizing the segmented OCTA and extracting the underlying vascular graph, which enables the quantitative assessment of various angioarchitectural properties, including individual vessel lengths and tortuosity. These tools, including the trained CNNs, are made publicly available as a user-friendly toolbox for researchers to input their OCTA images and subsequently receive the underlying vascular network graph with the associated angioarchitectural properties.     

深度学习重建算法优化能谱CT低单能量图像质量及检测肝脏低对比度小病灶能力

目的 观察深度学习重建(DLIR)算法用于优化能谱CT低单能量图像质量及提高检测肝脏低对比度小病灶能力的可行性。方法 纳入30例接受上腹部门脉期增强扫描的肝脏疾病患者,包括58个肝脏病灶,分别采用DLIR及基于混合模型的自适应统计迭代重建(ASIR-V)算法重建40~70 keV (间隔10 keV)单能量图像;根据肝脏、门静脉及肝脏病灶对比噪声比(CNR)和噪声进行主观评价,针对图像总体质量、病灶显著性和诊断信心评分进行主观评价,比较不同图像之间评价结果的差异。结果 相比ASIR-V图像,40~70 keV能级下,DLIR图像的CNR_肝脏、CNR_门静脉及CNR_肝脏病灶均显著增加而噪声均显著减少(P均<0.05);40~60 keV能级下,DLIR图像总体质量、病灶显著性及诊断信心评分均高于ASIR-V图像(P均<0.05)。结论 DLIR技术可显著减少低单能量成像噪声、改善图像质量并提高检测肝脏低对比度小病灶的能力。     

Fast and adaptive region merging based on perceptual hashing via multi-thresholding for SAR image segmentation

Due to the serious speckle noise in synthetic aperture radar (SAR) image, segmentation of SAR images is still a challenging problem. In this paper, a novel region merging method based on perceptual hashing is proposed for SAR image segmentation. In the proposed method, perceptual hash algorithm (PHA) is utilized to calculate the degree of similarity between different regions during region merging in SAR image segmentation. After reducing the speckle noise by Lee filter which maintains the sharpness of SAR image, a set of different homogeneous regions is constructed based on multi-thresholding and treated as the input data of region merging. The new contribution of this paper is the combination of multi-thresholding for initial segmentation and perceptual hash method for the adaptive process of region merging, which preserves the texture feature of input images and reduces the time complexity of the proposed method. The experimental results on synthetic and real SAR images show that the proposed algorithm is faster and attains higher-quality segmentation results than the three recent state-of-the-art image segmentation methods.     

Deep label fusion: A generalizable hybrid multi-atlas and deep convolutional neural network for medical image segmentation

Deep convolutional neural networks (DCNN) achieve very high accuracy in segmenting various anatomical structures in medical images but often suffer from relatively poor generalizability. Multi-atlas segmentation (MAS), while less accurate than DCNN in many applications, tends to generalize well to unseen datasets with different characteristics from the training dataset. Several groups have attempted to integrate the power of DCNN to learn complex data representations and the robustness of MAS to changes in image characteristics. However, these studies primarily focused on replacing individual components of MAS with DCNN models and reported marginal improvements in accuracy. In this study we describe and evaluate a 3D end-to-end hybrid MAS and DCNN segmentation pipeline, called Deep Label Fusion (DLF). The DLF pipeline consists of two main components with learnable weights, including a weighted voting subnet that mimics the MAS algorithm and a fine-tuning subnet that corrects residual segmentation errors to improve final segmentation accuracy. We evaluate DLF on five datasets that represent a diversity of anatomical structures (medial temporal lobe subregions and lumbar vertebrae) and imaging modalities (multi-modality, multi-field-strength MRI and Computational Tomography). These experiments show that DLF achieves comparable segmentation accuracy to nnU-Net (Isensee et al., 2020), the state-of-the-art DCNN pipeline, when evaluated on a dataset with similar characteristics to the training datasets, while outperforming nnU-Net on tasks that involve generalization to datasets with different characteristics (different MRI field strength or different patient population). DLF is also shown to consistently improve upon conventional MAS methods. In addition, a modality augmentation strategy tailored for multimodal imaging is proposed and demonstrated to be beneficial in improving the segmentation accuracy of learning-based methods, including DLF and DCNN, in missing data scenarios in test time as well as increasing the interpretability of the contribution of each individual modality.     

基于域对齐的深度学习方法在急性缺血性卒中的CT/MRI影像分割上的应用

目的  基于平扫CT提出一种域对齐方法来显著提高急性缺血性卒中(AIS)的早期快速诊断能力。方法  回顾性分析南方医科大学第三附属医院神经内科和神经外科2020年1月~2022年12月收治的入院后3 d内同时接受平扫头颅CT和MRI/DWI、ADC以及T2-Flair序列扫描的AIS患者,构建了一个由318例AIS病例组成的成对CT/MRI影像数据集,分别对每一组配对的教师-学生影像特征进行归一化;再以8∶2的比例随机分为训练集和验证集。设计一种新的生成性对抗性网络来对齐特征层上的跨模式输入,将细节丰富的MRI图像中的语义知识传递到CT图像中进行AIS分割,开发了一种新的域适应算法（Our DA）。结果  与目前性能表现较优异的医学影像分割模型nnUNet相比,Our DA明显优于nnU-Net,每一层验证集之间的分割精度提升约15%。结论  本研究构建的Our DA模型基于MRI/DWI序列的影像特征并迁移到平扫头颅CT上,对平扫头颅CT上的AIS病灶具有较高的自动分割性能,有助于早期自动识别AIS病灶。     

Automatic X-ray landmark detection and shape segmentation via data-driven joint estimation of image displacements

In this paper, we propose a new method for fully-automatic landmark detection and shape segmentation in X-ray images. To detect landmarks, we estimate the displacements from some randomly sampled image patches to the (unknown) landmark positions, and then we integrate these predictions via a voting scheme. Our key contribution is a new algorithm for estimating these displacements. Different from other methods where each image patch independently predicts its displacement, we jointly estimate the displacements from all patches together in a data driven way, by considering not only the training data but also geometric constraints on the test image. The displacements estimation is formulated as a convex optimization problem that can be solved efficiently. Finally, we use the sparse shape composition model as the a priori information to regularize the landmark positions and thus generate the segmented shape contour. We validate our method on X-ray image datasets of three different anatomical structures: complete femur, proximal femur and pelvis. Experiments show that our method is accurate and robust in landmark detection, and, combined with the shape model, gives a better or comparable performance in shape segmentation compared to state-of-the art methods. Finally, a preliminary study using CT data shows the extensibility of our method to 3D data.     

Comparison of chiropractic student scores before and after utilizing active learning techniques in a classroom setting

Objective

We report the differences in final examination scores achieved by students at the culmination of two different teaching strategies in an introductory skills course.

Methods

Multiple choice examination scores from six consecutive academic calendar sessions over 18 months (n = 503) were compared. Two groups were used: Cohort A (n = 290) represented students who were enrolled in the course 3 consecutive academic sessions before an instructional change and Cohort B (n = 213) included students who were enrolled in 3 consecutive academic sessions following the instructional change, which included a more active learning format. Statistical analyses used were 2-tailed independent t-test, one-way ANOVA, Tukey''s honestly significant difference (HSD), and effect size.

Results

The 2-tailed independent t-test revealed a significant difference between the two groups (t = −3.71, p < .001; 95% confidence interval [CI] 1.29–4.20). Significant difference was found in the highest performing subgroup compared to the lowest performing subgroup in Cohort A (F = 3.343, p = .037). For Cohort A subgroups 1 and 2, Tukey''s HSD was p < .028. In Cohort B, no difference was found among subgroups (F = 1.912, p = .150, HSD p > .105).

Conclusion

Compared to previous versions of the same course taught by the same instructor, the students in the new course design performed better, suggesting that using active learning techniques helps improve student achievement.Key Indexing Terms: Active Learning, Chiropractic, Education     

Predict brain MR image registration via sparse learning of appearance and transformation

We propose a new approach to register the subject image with the template by leveraging a set of intermediate images that are pre-aligned to the template. We argue that, if points in the subject and the intermediate images share similar local appearances, they may have common correspondence in the template. In this way, we learn the sparse representation of a certain subject point to reveal several similar candidate points in the intermediate images. Each selected intermediate candidate can bridge the correspondence from the subject point to the template space, thus predicting the transformation associated with the subject point at the confidence level that relates to the learned sparse coefficient. Following this strategy, we first predict transformations at selected key points, and retain multiple predictions on each key point, instead of allowing only a single correspondence. Then, by utilizing all key points and their predictions with varying confidences, we adaptively reconstruct the dense transformation field that warps the subject to the template. We further embed the prediction–reconstruction protocol above into a multi-resolution hierarchy. In the final, we refine our estimated transformation field via existing registration method in effective manners. We apply our method to registering brain MR images, and conclude that the proposed framework is competent to improve registration performances substantially.     

Deep learning facilitates fully automated brain image registration of optoacoustic tomography and magnetic resonance imaging

Multispectral optoacoustic tomography (MSOT) is an emerging optical imaging method providing multiplex molecular and functional information from the rodent brain. It can be greatly augmented by magnetic resonance imaging (MRI) which offers excellent soft-tissue contrast and high-resolution brain anatomy. Nevertheless, registration of MSOT-MRI images remains challenging, chiefly due to the entirely different image contrast rendered by these two modalities. Previously reported registration algorithms mostly relied on manual user-dependent brain segmentation, which compromised data interpretation and quantification. Here we propose a fully automated registration method for MSOT-MRI multimodal imaging empowered by deep learning. The automated workflow includes neural network-based image segmentation to generate suitable masks, which are subsequently registered using an additional neural network. The performance of the algorithm is showcased with datasets acquired by cross-sectional MSOT and high-field MRI preclinical scanners. The automated registration method is further validated with manual and half-automated registration, demonstrating its robustness and accuracy.     

基于V-Net卷积神经网络深度学习模型自动分割腰椎CT图像中的椎旁肌

目的 观察基于V-Net卷积神经网络(CNN)的深度学习(DL)模型自动分割腰椎CT图像中的椎旁肌的价值。方法 收集471例接受腰椎CT检查患者，按7∶3比例将其分为训练集(n=330)和测试集(n=141);采用2D V-Net进行训练，建立DL模型；观察其分割腰大肌、腰方肌、椎后肌群及椎旁肌的价值。结果 基于V-Net CNN的DL模型分割椎旁肌精度良好，戴斯相似系数(DSC)均较高、肌肉横截面积误差率(CSA error)均较低；其分割训练集图像中的腰大肌、腰方肌及椎旁肌的DSC均高于测试集(P均<0.05),而分割训练集中4组肌肉的CSA error均低于测试集(P均<0.05)。测试集内两两比较结果显示，该模型分割椎后肌群的DSC最高、腰方肌的DSC最低；分割腰方肌的CSA error最高、椎旁肌的CSA error最低(P均<0.05)。结论 以基于V-Net的DL模型自动分割椎旁肌的效能较佳。     

Eigenrank by committee: Von-Neumann entropy based data subset selection and failure prediction for deep learning based medical image segmentation

Manual delineation of anatomy on existing images is the basis of developing deep learning algorithms for medical image segmentation. However, manual segmentation is tedious. It is also expensive because clinician effort is necessary to ensure correctness of delineation. Consequently most algorithm development is based on a tiny fraction of the vast amount of imaging data collected at a medical center. Thus, selection of a subset of images from hospital databases for manual delineation - so that algorithms trained on such data are accurate and tolerant to variation, becomes an important challenge. We address this challenge using a novel algorithm. The proposed algorithm named ‘Eigenrank by Committee’ (EBC) first computes the degree of disagreement between segmentations generated by each DL model in a committee. Then, it iteratively adds to the committee, a DL model trained on cases where the disagreement is maximal. The disagreement between segmentations is quantified by the maximum eigenvalue of a Dice coefficient disagreement matrix a measure closely related to the Von Neumann entropy. We use EBC for selecting data subsets for manual labeling from a larger database of spinal canal segmentations as well as intervertebral disk segmentations. U-Nets trained on these subsets are used to generate segmentations on the remaining data. Similar sized data subsets are also randomly sampled from the respective databases, and U-Nets are trained on these random subsets as well. We found that U-Nets trained using data subsets selected by EBC, generate segmentations with higher average Dice coefficients on the rest of the database than U-Nets trained using random sampling (p < 0.05 using t-tests comparing averages). Furthermore, U-Nets trained using data subsets selected by EBC generate segmentations with a distribution of Dice coefficients that demonstrate significantly (p < 0.05 using Bartlett’s test) lower variance in comparison to U-Nets trained using random sampling for all datasets. We believe that this lower variance indicates that U-Nets trained with EBC are more robust than U-Nets trained with random sampling.     

Histogram of Oriented Gradients meet deep learning: A novel multi-task deep network for 2D surgical image semantic segmentation

We present our novel deep multi-task learning method for medical image segmentation. Existing multi-task methods demand ground truth annotations for both the primary and auxiliary tasks. Contrary to it, we propose to generate the pseudo-labels of an auxiliary task in an unsupervised manner. To generate the pseudo-labels, we leverage Histogram of Oriented Gradients (HOGs), one of the most widely used and powerful hand-crafted features for detection. Together with the ground truth semantic segmentation masks for the primary task and pseudo-labels for the auxiliary task, we learn the parameters of the deep network to minimize the loss of both the primary task and the auxiliary task jointly. We employed our method on two powerful and widely used semantic segmentation networks: UNet and U2Net to train in a multi-task setup. To validate our hypothesis, we performed experiments on two different medical image segmentation data sets. From the extensive quantitative and qualitative results, we observe that our method consistently improves the performance compared to the counter-part method. Moreover, our method is the winner of FetReg Endovis Sub-challenge on Semantic Segmentation organised in conjunction with MICCAI 2021. Code and implementation details are available at:https://github.com/thetna/medical_image_segmentation.     

Interpretable deep learning systems for multi-class segmentation and classification of non-melanoma skin cancer

We apply for the first-time interpretable deep learning methods simultaneously to the most common skin cancers (basal cell carcinoma, squamous cell carcinoma and intraepidermal carcinoma) in a histological setting. As these three cancer types constitute more than 90% of diagnoses, we demonstrate that the majority of dermatopathology work is amenable to automatic machine analysis. A major feature of this work is characterising the tissue by classifying it into 12 meaningful dermatological classes, including hair follicles, sweat glands as well as identifying the well-defined stratified layers of the skin. These provide highly interpretable outputs as the network is trained to represent the problem domain in the same way a pathologist would. While this enables a high accuracy of whole image classification (93.6-97.9%), by characterising the full context of the tissue we can also work towards performing routine pathologist tasks, for instance, orientating sections and automatically assessing and measuring surgical margins. This work seeks to inform ways in which future computer aided diagnosis systems could be applied usefully in a clinical setting with human interpretable outcomes.