Deep probabilistic tracking of particles in fluorescence microscopy images

Tracking of particles in temporal fluorescence microscopy image sequences is of fundamental importance to quantify dynamic processes of intracellular structures as well as virus structures. We introduce a probabilistic deep learning approach for fluorescent particle tracking, which is based on a recurrent neural network that mimics classical Bayesian filtering. Compared to previous deep learning methods for particle tracking, our approach takes into account uncertainty, both aleatoric and epistemic uncertainty. Thus, information about the reliability of the computed trajectories is determined. Manual tuning of tracking parameters is not necessary and prior knowledge about the noise statistics is not required. Short and long-term temporal dependencies of individual object dynamics are exploited for state prediction, and assigned detections are used to update the predicted states. For correspondence finding, we introduce a neural network which computes assignment probabilities jointly across multiple detections as well as determines the probabilities of missing detections. Training requires only simulated data and therefore tedious manual annotation of ground truth is not needed. We performed a quantitative performance evaluation based on synthetic and real 2D as well as 3D fluorescence microscopy images. We used image data of the Particle Tracking Challenge as well as real time-lapse fluorescence microscopy images displaying virus structures and chromatin structures. It turned out that our approach yields state-of-the-art results or improves the tracking results compared to previous methods.     

DCAN: Deep contour-aware networks for object instance segmentation from histology images

In histopathological image analysis, the morphology of histological structures, such as glands and nuclei, has been routinely adopted by pathologists to assess the malignancy degree of adenocarcinomas. Accurate detection and segmentation of these objects of interest from histology images is an essential prerequisite to obtain reliable morphological statistics for quantitative diagnosis. While manual annotation is error-prone, time-consuming and operator-dependant, automated detection and segmentation of objects of interest from histology images can be very challenging due to the large appearance variation, existence of strong mimics, and serious degeneration of histological structures. In order to meet these challenges, we propose a novel deep contour-aware network (DCAN) under a unified multi-task learning framework for more accurate detection and segmentation. In the proposed network, multi-level contextual features are explored based on an end-to-end fully convolutional network (FCN) to deal with the large appearance variation. We further propose to employ an auxiliary supervision mechanism to overcome the problem of vanishing gradients when training such a deep network. More importantly, our network can not only output accurate probability maps of histological objects, but also depict clear contours simultaneously for separating clustered object instances, which further boosts the segmentation performance. Our method ranked the first in two histological object segmentation challenges, including 2015 MICCAI Gland Segmentation Challenge and 2015 MICCAI Nuclei Segmentation Challenge. Extensive experiments on these two challenging datasets demonstrate the superior performance of our method, surpassing all the other methods by a significant margin.     

Accurate non-maximum suppression for object detection in high-resolution remote sensing images

Non-maximum suppression (NMS) is widely adopted as a post-processing step in the state-of-the-art object detection pipelines to merge the nearby detections around one object. However, its performance is affected by objects that are highly overlapped with each other, and its localization accuracy depends solely on the highest scored detection. To tackle this, an accurate NMS method is proposed in this letter, which gradually merges the highly overlapped detections in an iterative way. In each iteration, detections overlapped with the highest scored one are grouped with a harder threshold to regress for a new proposal, and then the scores within the group are softly suppressed. This process is recursively applied on the remaining detections. The proposed method can not only detect more overlapped objects, but also achieve better object localization accuracy. Experimental results demonstrate that this simple and unsupervised method can gain obvious performance improvement on the majority of classes, compared with the state-of-the-art NMS methods.     

An algorithm to improve the detection of ocean fronts from whiskbroom scanner images

High-resolution satellite imagery is a valuable data source to analyse ocean submesoscale dynamics (i.e., with spatial scales of the order of 1–10 km) and investigate their impact on turbulent mixing, energetics of mesoscale vortices, instability processes or phytoplankton blooms. However, data acquired by satellite sensors often suffer from instrumental noise that degrades image quality and therefore compromises the detection of ocean fronts as well as the estimation of its physical characteristics. A well-known artefact in data characteristic of whiskbroom scanners is stripe noise. In this article, we propose an algorithm that improves the detection of ocean fronts by removing the impact of striping on the observed gradient field. We use level 2 sea surface temperature and chlorophyll-a products derived from NASA’s Moderate Resolution Imaging Spectroradiometer to illustrate the algorithm performance.     

DeepDistance: A multi-task deep regression model for cell detection in inverted microscopy images

This paper presents a new deep regression model, which we call DeepDistance, for cell detection in images acquired with inverted microscopy. This model considers cell detection as a task of finding most probable locations that suggest cell centers in an image. It represents this main task with a regression task of learning an inner distance metric. However, different than the previously reported regression based methods, the DeepDistance model proposes to approach its learning as a multi-task regression problem where multiple tasks are learned by using shared feature representations. To this end, it defines a secondary metric, normalized outer distance, to represent a different aspect of the problem and proposes to define its learning as complementary to the main cell detection task. In order to learn these two complementary tasks more effectively, the DeepDistance model designs a fully convolutional network (FCN) with a shared encoder path and end-to-end trains this FCN to concurrently learn the tasks in parallel. For further performance improvement on the main task, this paper also presents an extended version of the DeepDistance model that includes an auxiliary classification task and learns it in parallel to the two regression tasks by also sharing feature representations with them. DeepDistance uses the inner distances estimated by these FCNs in a detection algorithm to locate individual cells in a given image. In addition to this detection algorithm, this paper also suggests a cell segmentation algorithm that employs the estimated maps to find cell boundaries. Our experiments on three different human cell lines reveal that the proposed multi-task learning models, the DeepDistance model and its extended version, successfully identify the locations of cell as well as delineate their boundaries, even for the cell line that was not used in training, and improve the results of its counterparts.     

Integrating watersheds and critical point analysis for object detection in discrete 2D images

This paper presents an improved method for the detection of "significant" low-level objects in medical images. The method overcomes topological problems where multiple redundant saddle points are detected in digital images. Information derived from watershed regions is used to select and refine saddle points in the discrete domain and to construct the watersheds and watercourses (ridges and valleys). We also demonstrate an improved method of pruning the tessellation by which to define low level objects in zero order images. The algorithm was applied on a set of medical images with promising results. Evaluation was based on theoretical analysis and human observer experiments.     

Deep learning for bone marrow cell detection and classification on whole-slide images

Bone marrow (BM) examination is an essential step in both diagnosing and managing numerous hematologic disorders. BM nucleated differential count (NDC) analysis, as part of BM examination, holds the most fundamental and crucial information. However, there are many challenges to perform automated BM NDC analysis on whole-slide images (WSIs), including large dimensions of data to process, complicated cell types with subtle differences. To the authors best knowledge, this is the first study on fully automatic BM NDC using WSIs with 40x objective magnification, which can replace traditional manual counting relying on light microscopy via oil-immersion 100x objective lens with a total 1000x magnification. In this study, we develop an efficient and fully automatic hierarchical deep learning framework for BM NDC WSI analysis in seconds. The proposed hierarchical framework consists of (1) a deep learning model for rapid localization of BM particles and cellular trails generating regions of interest (ROI) for further analysis, (2) a patch-based deep learning model for cell identification of 16 cell types, including megakaryocytes, mitotic cells, and four stages of erythroblasts which have not been demonstrated in previous studies before, and (3) a fast stitching model for integrating patch-based results and producing final outputs. In evaluation, the proposed method is firstly tested on a dataset with a total of 12,426 annotated cells using cross validation, achieving high recall and accuracy of 0.905 ± 0.078 and 0.989 ± 0.006, respectively, and taking only 44 seconds to perform BM NDC analysis for a WSI. To further examine the generalizability of our model, we conduct an evaluation on the second independent dataset with a total of 3005 cells, and the results show that the proposed method also obtains high recall and accuracy of 0.842 and 0.988, respectively. In comparison with the existing small-image-based benchmark methods, the proposed method demonstrates superior performance in recall, accuracy and computational time.     

Deep learning approach to improve tangential resolution in photoacoustic tomography

In circular scan photoacoustic tomography (PAT), the axial resolution is spatially invariant and is limited by the bandwidth of the detector. However, the tangential resolution is spatially variant and is dependent on the aperture size of the detector. In particular, the tangential resolution improves with the decreasing aperture size. However, using a detector with a smaller aperture reduces the sensitivity of the transducer. Thus, large aperture size detectors are widely preferred in circular scan PAT imaging systems. Although several techniques have been proposed to improve the tangential resolution, they have inherent limitations such as high cost and the need for customized detectors. Herein, we propose a novel deep learning architecture to counter the spatially variant tangential resolution in circular scanning PAT imaging systems. We used a fully dense U-Net based convolutional neural network architecture along with 9 residual blocks to improve the tangential resolution of the PAT images. The network was trained on the simulated datasets and its performance was verified by experimental in vivo imaging. Results show that the proposed deep learning network improves the tangential resolution by eight folds, without compromising the structural similarity and quality of image.     

How to improve the teaching of urine microscopy.

BACKGROUND: Urinary microscopy is difficult to teach. This paper describes a 1-day course on urine microscopy, which was based on both theoretical and practical sessions at the microscope, during which real urine samples were examined. METHODS: The course was based on: a) an introductory presentation with slides on the main components of urinary sediments and their clinical correlates; b) examination of fixed urine samples under the guidance of two experts; and c) the use of two microscopes, each equipped with a co-observation device for up to 15 observers. RESULTS: Throughout 2005, four courses were held in four Italian towns. Altogether, there were 97 participants (20-27 per course) from 75 laboratories, all graduates with wide but variable experience in the field. During each course, 17-22 urinary sediment components were shown by both bright-field and phase-contrast microscopy and, when indicated, by polarized light. Tests set before and after the course showed a significant improvement (p<0.01) in the identification of erythrocyte subtypes, epithelial cells, fatty components, various types of casts and drug crystals. A questionnaire conducted with participants by phone several months after the course demonstrated that 51.6% and 32.3% of laboratories have introduced or formally requested phase-contrast and polarized-light microscopy, respectively; 45.2% have changed the terminology for urinary epithelial cells; and 87.1% have identified for the first time urinary sediment components that were not recognized or not considered before the course. CONCLUSIONS: Our course demonstrates that it is possible to improve the teaching of urinary microscopy.     

Tripartite-GAN: Synthesizing liver contrast-enhanced MRI to improve tumor detection

Contrast-enhanced magnetic resonance imaging (CEMRI) is crucial for the diagnosis of patients with liver tumors, especially for the detection of benign tumors and malignant tumors. However, it suffers from high-risk, time-consuming, and expensive in current clinical diagnosis due to the use of the gadolinium-based contrast agent (CA) injection. If the CEMRI can be synthesized without CA injection, there is no doubt that it will greatly optimize the diagnosis. In this study, we propose a Tripartite Generative Adversarial Network (Tripartite-GAN) as a non-invasive, time-saving, and inexpensive clinical tool by synthesizing CEMRI to detect tumors without CA injection. Specifically, our innovative Tripartite-GAN combines three associated-networks (an attention-aware generator, a convolutional neural network-based discriminator, and a region-based convolutional neural network-based detector) for the first time, which achieves CEMRI synthesis and tumor detection promoting each other in an end-to-end framework. The generator facilitates detector for accurate tumor detection via synthesizing tumor-specific CEMRI. The detector promotes the generator for accurate CEMRI synthesis via the back-propagation. In order to synthesize CEMRI of equivalent clinical value to real CEMRI, the attention-aware generator expands the receptive field via hybrid convolution, and enhances feature representation and context learning of multi-class liver MRI via dual attention mechanism, and improves the performance of convergence of loss via residual learning. Moreover, the attention maps obtained from the generator newly added into the detector improve the performance of tumor detection. The discriminator promotes the generator to synthesize high-quality CEMRI via the adversarial learning strategy. This framework is tested on a large corpus of axial T1 FS Pre-Contrast MRI and axial T1 FS Delay MRI of 265 subjects. Experimental results and quantitative evaluation demonstrate that the Tripartite-GAN achieves high-quality CEMRI synthesis that peak signal-to-noise rate of 28.8 and accurate tumor detection that accuracy of 89.4%, which reveals that Tripartite-GAN can aid in the clinical diagnosis of liver tumors.     

Axial scanning of dual focus to improve light sheet microscopy

Axially swept light sheet microscopy (ASLM) is an emerging technique that enables isotropic, subcellular resolution imaging with high optical sectioning capability over a large field-of-view (FOV). Due to its versatility across a broad range of immersion media, it has been utilized to image specimens that may range from live cells to intact chemically cleared organs. However, because of its design, the performance of ASLM-based microscopes is impeded by a low detection signal and the maximum achievable frame-rate for full FOV imaging. Here we present a new optical concept that pushes the limits of ASLM further by scanning two staggered light sheets and simultaneously synchronizing the rolling shutter of a scientific camera. For a particular peak-illumination-intensity, this idea can make ASLMs image twice as fast without compromising the detection signal. Alternately, for a particular frame rate our method doubles the detection signal without requiring to double the peak-illumination-power, thereby offering a gentler illumination scheme compared to tradition single-focus ASLM. We demonstrate the performance of our instrument by imaging fluorescent beads and a PEGASOS cleared-tissue mouse brain.     

Denoising multiplexed microscopy images in n-dimensional spectral space

Hyperspectral fluorescence microscopy images of biological specimens frequently contain multiple observations of a sparse set of spectral features spread in space with varying intensity. Here, we introduce a spectral vector denoising algorithm that filters out noise without sacrificing spatial information by leveraging redundant observations of spectral signatures. The algorithm applies an n-dimensional Chebyshev or Fourier transform to cluster pixels based on spectral similarity independent of pixel intensity or location, and a denoising convolution filter is then applied in this spectral space. The denoised image may then undergo spectral decomposition analysis with enhanced accuracy. Tests utilizing both simulated and empirical microscopy data indicate that denoising in 3 to 5-dimensional (3D to 5D) spectral spaces decreases unmixing error by up to 70% without degrading spatial resolution.     

Phase detection in acoustic microscopy

The scanning laser acoustic microscope provides a relatively simple means by which speed of sound data can be collected from biological specimens. By employing a phase detection circuit, phase-contour lines can be superimposed on acoustic micrographs and digitized for direct speed of sound calculations. Because of the existence of mild field non-uniformities, the phase reference must be mathematically modeled. This is accomplished by employing a polynomial function and simple linear regression. Scattering can be studied by using a frequency-selective "dark field" approach. Speed of sound maps are readily produced from the phase data by applying the simple geometric relationships of wave propagation; however the theoretical limitations of the technique must be kept in mind. Using this approach, useful data have been obtained.     

Discriminative generalized Hough transform for object localization in medical images

Purpose   This paper proposes the discriminative generalized Hough transform (DGHT) as an efficient and reliable means for object localization in medical images. It is meant to give a deeper insight into the underlying theory and a comprehensive overview of the methodology and the scope of applications. Methods   The DGHT combines the generalized Hough transform (GHT) with a discriminative training technique for the GHT models to obtain more efficient and robust localization results. To this end, the model points are equipped with individual weights, which are trained discriminatively with respect to a minimal localization error. Through this weighting, the models become more robust since the training focuses on common features of the target object over a set of training images. Unlike other weighting strategies, our training algorithm focuses on the error rate and allows for negative weights, which can be employed to encode rivaling structures into the model. The basic algorithm is presented here in conjunction with several extensions for fully automatic and faster processing. These include: (1) the automatic generation of models from training images and their iterative refinement, (2) the training of joint models for similar objects, and (3) a multi-level approach. Results   The algorithm is tested successfully for the knee in long-leg radiographs (97.6 % success rate), the vertebrae in C-arm CT (95.5 % success rate), and the femoral head in whole-body MR (100 % success rate). In addition, it is compared to Hough forests (Gall et al. in IEEE Trans Pattern Anal Mach Intell 33(11):2188–2202, 2011) for the task of knee localization (97.8 % success rate). Conclusion   The DGHT has proven to be a general procedure, which can be easily applied to various tasks with high success rates.     

Interval change analysis to improve computer aided detection in mammography

We are developing computer aided diagnosis (CAD) techniques to study interval changes between two consecutive mammographic screening rounds. We have previously developed methods for the detection of malignant masses based on features extracted from single mammographic views. The goal of the present work was to improve our detection method by including temporal information in the CAD program. Toward this goal, we have developed a regional registration technique. This technique links a suspicious location on the current mammogram with a corresponding location on the prior mammogram. The novelty of our method is that the search for correspondence is done in feature space. This has the advantage that very small lesions and architectural distortions may be found as well. Following the linking process several features are calculated for the current and prior region. Temporal features are obtained by combining the feature values from both regions. We evaluated the detection performance with and without the use of temporal features on a data set containing 2873 temporal film pairs from 938 patients. There were 589 cases in which the current mammogram contained exactly one malignant mass. Cross validation was used to partition the data set into a train set and a test set. The train set was used for feature selection and classifier training, the test set for classifier evaluation. FROC (free response operating characteristic) analysis showed an improvement in detection performance with the use of temporal features.     

Deep learning automated pathology in ex vivo microscopy

Standard histopathology is currently the gold standard for assessment of margin status in Mohs surgical removal of skin cancer. Ex vivo confocal microscopy (XVM) is potentially faster, less costly and inherently 3D/digital compared to standard histopathology. Despite these advantages, XVM use is not widespread due, in part, to the need for pathologists to retrain to interpret XVM images. We developed artificial intelligence (AI)-driven XVM pathology by implementing algorithms that render intuitive XVM pathology images identical to standard histopathology and produce automated tumor positivity maps. XVM images have fluorescence labeling of cellular and nuclear biology on the background of endogenous (unstained) reflectance contrast as a grounding counter-contrast. XVM images of 26 surgical excision specimens discarded after Mohs micrographic surgery were used to develop an XVM data pipeline with 4 stages: flattening, colorizing, enhancement and automated diagnosis. The first two stages were novel, deterministic image processing algorithms, and the second two were AI algorithms. Diagnostic sensitivity and specificity were calculated for basal cell carcinoma detection as proof of principal for the XVM image processing pipeline. The resulting diagnostic readouts mimicked the appearance of histopathology and found tumor positivity that required first collapsing the confocal stack to a 2D image optimized for cellular fluorescence contrast, then a dark field-to-bright field colorizing transformation, then either an AI image transformation for visual inspection or an AI diagnostic binary image segmentation of tumor obtaining a diagnostic sensitivity and specificity of 88% and 91% respectively. These results show that video-assisted micrographic XVM pathology could feasibly aid margin status determination in micrographic surgery of skin cancer.     

Wavelet-based edge detection in ultrasound images

We introduce a new wavelet-based method for edge detection in ultrasound (US) images. Each beam that is analyzed is first transformed into the wavelet domain using the continuous wavelet transform (CWT). Because the CWT preserves both scale and time information, it is possible to separate the signal into a number of scales. The edge is localized by first determining the scale at which the power spectrum, based on the wavelet transform, has its maximum value. Next, at this scale we find the position of the peak for the squared CWT. This method does not depend on any threshold, after the range of scales have been determined. We suggest a range of scales for US images in general. Sample edge detections are demonstrated in US images of straight and jagged edges of simple structures submerged in water bath, and of an abdominal aorta aneurysm phantom.     

Seeing the invisible in differential interference contrast microscopy images

Automated microscopy image restoration, especially in Differential Interference Contrast (DIC) imaging modality, has attracted increasing attentions since it greatly facilitates long-term living cell analysis without staining. Although the previous work on DIC image restoration is able to restore the nuclei regions of living cells, it is still challenging to reconstruct the unnoticeable cytoplasm details in DIC images. In this paper, we propose to extract the tiny movement information of living cells in DIC images and reveal the hidden details in DIC images by magnifying the cells’ motion as well as attenuating the intensity variation from the background. From our restored images, we can clearly observe the previously-invisible details in DIC images. Experiments on two DIC image datasets show that the motion-based restoration method can reveal the hidden details of living cells. In addition, we demonstrate our restoration method can also be applied to other imaging modalities such as the phase contrast microscopy to enhance cells’ details. Furthermore, based on the pixel-level restoration results, we can obtain the object-level segmentation by leveraging a label propagation approach, providing promising results on facilitating the cell shape and behavior analysis. The proposed algorithm can be a software module to enhance the visualization capability of microscopes.     

Understanding the phase contrast optics to restore artifact-free microscopy images for segmentation

Phase contrast, a noninvasive microscopy imaging technique, is widely used to capture time-lapse images to monitor the behavior of transparent cells without staining or altering them. Due to the optical principle, phase contrast microscopy images contain artifacts such as the halo and shade-off that hinder image segmentation, a critical step in automated microscopy image analysis. Rather than treating phase contrast microscopy images as general natural images and applying generic image processing techniques on them, we propose to study the optical properties of the phase contrast microscope to model its image formation process. The phase contrast imaging system can be approximated by a linear imaging model. Based on this model and input image properties, we formulate a regularized quadratic cost function to restore artifact-free phase contrast images that directly correspond to the specimen's optical path length. With artifacts removed, high quality segmentation can be achieved by simply thresholding the restored images. The imaging model and restoration method are quantitatively evaluated on microscopy image sequences with thousands of cells captured over several days. We also demonstrate that accurate restoration lays the foundation for high performance in cell detection and tracking.