Automated segmentation and tracking of non-rigid objects in time-lapse microscopy videos of polymorphonuclear neutrophils

Time-lapse microscopy is an important technique to study the dynamics of various biological processes. The labor-intensive manual analysis of microscopy videos is increasingly replaced by automated segmentation and tracking methods. These methods are often limited to certain cell morphologies and/or cell stainings. In this paper, we present an automated segmentation and tracking framework that does not have these restrictions. In particular, our framework handles highly variable cell shapes and does not rely on any cell stainings. Our segmentation approach is based on a combination of spatial and temporal image variations to detect moving cells in microscopy videos. This method yields a sensitivity of 99% and a precision of 95% in object detection. The tracking of cells consists of different steps, starting from single-cell tracking based on a nearest-neighbor-approach, detection of cell–cell interactions and splitting of cell clusters, and finally combining tracklets using methods from graph theory. The segmentation and tracking framework was applied to synthetic as well as experimental datasets with varying cell densities implying different numbers of cell–cell interactions. We established a validation framework to measure the performance of our tracking technique. The cell tracking accuracy was found to be >99% for all datasets indicating a high accuracy for connecting the detected cells between different time points.     

Context aware spatio-temporal cell tracking in densely packed multilayer tissues

Modern live imaging technique enables us to observe the internal part of a tissue over time by generating serial optical images containing spatio-temporal slices of hundreds of tightly packed cells. Automated tracking of plant and animal cells from such time lapse live-imaging datasets of a developing multicellular tissue is required for quantitative, high throughput analysis of cell division, migration and cell growth. In this paper, we present a novel cell tracking method that exploits the tight spatial topology of neighboring cells in a multicellular field as contextual information and combines it with physical features of individual cells for generating reliable cell lineages. The 2D image slices of multicellular tissues are modeled as a conditional random field and pairwise cell to cell similarities are obtained by estimating marginal probability distributions through loopy belief propagation on this CRF. These similarity scores are further used in a spatio-temporal graph labeling problem to obtain the optimal and feasible set of correspondences between individual cell slices across the 4D image dataset. We present results on (3D + t) confocal image stacks of Arabidopsis shoot meristem and show that the method is capable of handling many visual analysis challenges associated with such cell tracking problems, viz. poor feature quality of individual cells, low SNR in parts of images, variable number of cells across slices and cell division detection.     

Fast automatic myocardial segmentation in 4D cine CMR datasets

A novel automatic 3D+time left ventricle (LV) segmentation framework is proposed for cardiac magnetic resonance (CMR) datasets. The proposed framework consists of three conceptual blocks to delineate both endo and epicardial contours throughout the cardiac cycle: (1) an automatic 2D mid-ventricular initialization and segmentation; (2) an automatic stack initialization followed by a 3D segmentation at the end-diastolic phase; and (3) a tracking procedure. Hereto, we propose to adapt the recent B-spline Explicit Active Surfaces (BEAS) framework to the properties of CMR images by integrating dedicated energy terms. Moreover, we extend the coupled BEAS formalism towards its application in 3D MR data by adapting it to a cylindrical space suited to deal with the topology of the image data. Furthermore, a fast stack initialization method is presented for efficient initialization and to enforce consistent cylindrical topology. Finally, we make use of an anatomically constrained optical flow method for temporal tracking of the LV surface.The proposed framework has been validated on 45 CMR datasets taken from the 2009 MICCAI LV segmentation challenge. Results show the robustness, efficiency and competitiveness of the proposed method both in terms of accuracy and computational load.     

Spontaneous hepatocyte migration towards an endothelial cell tube network

A crucial part of the engineering liver tissue is contribution of nonparenchymal cells and maintenance of a complex three‐dimensional (3D) structure in vitro for their normal physiology and function. We generated 3D hepatic tissue using primary isolated rat hepatocytes and an endothelial cell tube network from human endothelial vein epithelial cells (HUVECs). To create the 3D hepatic tissue, coculture of primary hepatocytes and tube‐structured HUVECs was performed on a Matrigel®. After the HUVECs formed the tube structures, primary isolated rat hepatocytes were inoculated onto the HUVEC tube‐structured layer and cultured for 24 hr. We investigated the cell migration, cellular interaction, and distributions of HUVEC tube structures and hepatocytes using multi cell‐imaging incubator, confocal microscopy, and electron microscopy analyses. During the culture time, time‐lapse imaging showed spontaneous migration of the hepatocytes in the gel, and after the 24‐hr culture period, the vast majority of the hepatocytes had moved and adhered to the surface of the HUVEC tube structures. A confocal microscopy assay confirmed this unique 3D cellular interaction between hepatocytes and HUVEC tube structures. The hepatocytes were able to maintain their spherical shape, as well as HUVECs (tube‐like form with tubular cavity). We speculate that coculturing of hepatocytes and endothelial cells replicates part of their normal physiology and may help induce migration in vitro and the growth of complex biological tissue structures.     

Fully automatic segmentation of the mitral leaflets in 3D transesophageal echocardiographic images using multi-atlas joint label fusion and deformable medial modeling

Comprehensive visual and quantitative analysis of in vivo human mitral valve morphology is central to the diagnosis and surgical treatment of mitral valve disease. Real-time 3D transesophageal echocardiography (3D TEE) is a practical, highly informative imaging modality for examining the mitral valve in a clinical setting. To facilitate visual and quantitative 3D TEE image analysis, we describe a fully automated method for segmenting the mitral leaflets in 3D TEE image data. The algorithm integrates complementary probabilistic segmentation and shape modeling techniques (multi-atlas joint label fusion and deformable modeling with continuous medial representation) to automatically generate 3D geometric models of the mitral leaflets from 3D TEE image data. These models are unique in that they establish a shape-based coordinate system on the valves of different subjects and represent the leaflets volumetrically, as structures with locally varying thickness. In this work, expert image analysis is the gold standard for evaluating automatic segmentation. Without any user interaction, we demonstrate that the automatic segmentation method accurately captures patient-specific leaflet geometry at both systole and diastole in 3D TEE data acquired from a mixed population of subjects with normal valve morphology and mitral valve disease.     

Optimized U-Net model for 3D light-sheet image segmentation of zebrafish trunk vessels

The growth of zebrafish''s vessels can be used as an indicator of the vascular development process and to study the biological mechanisms. The three-dimensional (3D) structures of zebrafish''s trunk vessels could be imaged by state-of-art light-sheet fluorescent microscopy with high efficiency. A large amount of data was then produced. Accurate segmentation of these 3D images becomes a new bottleneck for automatic and quantitative analysis. Here, we propose a Multi-scale 3D U-Net model to perform the segmentation of trunk vessels. The segmentation accuracies of 82.3% and 83.0%, as evaluated by the IoU (Intersection over Union) parameter, were achieved for intersegmental vessels and the dorsal longitudinal anastomotic vessels respectively. The growth of zebrafish vasculature from 42-62 hours was then analyzed quantitatively.     

Fully automatic three-dimensional visualization of intravascular optical coherence tomography images: methods and feasibility in vivo

Intravascular optical coherence tomography (IV-OCT) is an imaging modality that can be used for the assessment of intracoronary stents. Recent publications pointed to the fact that 3D visualizations have potential advantages compared to conventional 2D representations. However, 3D imaging still requires a time consuming manual procedure not suitable for on-line application during coronary interventions. We propose an algorithm for a rapid and fully automatic 3D visualization of IV-OCT pullbacks. IV-OCT images are first processed for the segmentation of the different structures. This also allows for automatic pullback calibration. Then, according to the segmentation results, different structures are depicted with different colors to visualize the vessel wall, the stent and the guide-wire in details. Final 3D rendering results are obtained through the use of a commercial 3D DICOM viewer. Manual analysis was used as ground-truth for the validation of the segmentation algorithms. A correlation value of 0.99 and good limits of agreement (Bland Altman statistics) were found over 250 images randomly extracted from 25 in vivo pullbacks. Moreover, 3D rendering was compared to angiography, pictures of deployed stents made available by the manufacturers and to conventional 2D imaging corroborating visualization results. Computational time for the visualization of an entire data sets resulted to be ~74 sec. The proposed method allows for the on-line use of 3D IV-OCT during percutaneous coronary interventions, potentially allowing treatments optimization.OCIS codes: (170.4500) Optical coherence tomography, (100.6890) Three-dimensional image processing, (330.5000) Vision - patterns and recognition     

A review of 3D vessel lumen segmentation techniques: Models,features and extraction schemes

Vascular diseases are among the most important public health problems in developed countries. Given the size and complexity of modern angiographic acquisitions, segmentation is a key step toward the accurate visualization, diagnosis and quantification of vascular pathologies.Despite the tremendous amount of past and on-going dedicated research, vascular segmentation remains a challenging task. In this paper, we review state-of-the-art literature on vascular segmentation, with a particular focus on 3D contrast-enhanced imaging modalities (MRA and CTA). We structure our analysis along three axes: models, features and extraction schemes. We first detail model-based assumptions on the vessel appearance and geometry which can embedded in a segmentation approach. We then review the image features that can be extracted to evaluate these models. Finally, we discuss how existing extraction schemes combine model and feature information to perform the segmentation task.Each component (model, feature and extraction scheme) plays a crucial role toward the efficient, robust and accurate segmentation of vessels of interest. Along each axis of study, we discuss the theoretical and practical properties of recent approaches and highlight the most advanced and promising ones.     

Understanding the phase contrast optics to restore artifact-free microscopy images for segmentation

Phase contrast, a noninvasive microscopy imaging technique, is widely used to capture time-lapse images to monitor the behavior of transparent cells without staining or altering them. Due to the optical principle, phase contrast microscopy images contain artifacts such as the halo and shade-off that hinder image segmentation, a critical step in automated microscopy image analysis. Rather than treating phase contrast microscopy images as general natural images and applying generic image processing techniques on them, we propose to study the optical properties of the phase contrast microscope to model its image formation process. The phase contrast imaging system can be approximated by a linear imaging model. Based on this model and input image properties, we formulate a regularized quadratic cost function to restore artifact-free phase contrast images that directly correspond to the specimen's optical path length. With artifacts removed, high quality segmentation can be achieved by simply thresholding the restored images. The imaging model and restoration method are quantitatively evaluated on microscopy image sequences with thousands of cells captured over several days. We also demonstrate that accurate restoration lays the foundation for high performance in cell detection and tracking.     

Optical clearing based cellular-level 3D visualization of intact lymph node cortex

Lymph node (LN) is an important immune organ that controls adaptive immune responses against foreign pathogens and abnormal cells. To facilitate efficient immune function, LN has highly organized 3D cellular structures, vascular and lymphatic system. Unfortunately, conventional histological analysis relying on thin-sliced tissue has limitations in 3D cellular analysis due to structural disruption and tissue loss in the processes of fixation and tissue slicing. Optical sectioning confocal microscopy has been utilized to analyze 3D structure of intact LN tissue without physical tissue slicing. However, light scattering within biological tissues limits the imaging depth only to superficial portion of LN cortex. Recently, optical clearing techniques have shown enhancement of imaging depth in various biological tissues, but their efficacy for LN are remained to be investigated. In this work, we established optical clearing procedure for LN and achieved 3D volumetric visualization of the whole cortex of LN. More than 4 times improvement in imaging depth was confirmed by using LN obtained from H2B-GFP/actin-DsRed double reporter transgenic mouse. With adoptive transfer of GFP expressing B cells and DsRed expressing T cells and fluorescent vascular labeling by anti-CD31 and anti-LYVE-1 antibody conjugates, we successfully visualized major cellular-level structures such as T-cell zone, B-cell follicle and germinal center. Further, we visualized the GFP expressing metastatic melanoma cell colony, vasculature and lymphatic vessels in the LN cortex.OCIS codes: (170.3880) Medical and biological imaging, (170.1790) Confocal microscopy, (170.3660) Light propagation in tissues, (170.6935) Tissue characterization     

Mesoscopic in vivo 3-D tracking of sparse cell populations using angular multiplexed optical projection tomography

We describe an angular multiplexed imaging technique for 3-D in vivo cell tracking of sparse cell distributions and optical projection tomography (OPT) with superior time-lapse resolution and a significantly reduced light dose compared to volumetric time-lapse techniques. We demonstrate that using dual axis OPT, where two images are acquired simultaneously at different projection angles, can enable localization and tracking of features in 3-D with a time resolution equal to the camera frame rate. This is achieved with a 200x reduction in light dose compared to an equivalent volumetric time-lapse single camera OPT acquisition with 200 projection angles. We demonstrate the application of this technique to mapping the 3-D neutrophil migration pattern observed over ~25.5 minutes in a live 2 day post-fertilisation transgenic LysC:GFP zebrafish embryo following a tail wound.OCIS codes: (170.2520) Fluorescence microscopy, (170.6900) Three-dimensional microscopy, (170.6920) Time-resolved imaging, (170.3010) Image reconstruction techniques     

Spatio-temporal cell cycle phase analysis using level sets and fast marching methods

Enabled by novel molecular markers, fluorescence microscopy enables the monitoring of multiple cellular functions using live cell assays. Automated image analysis is necessary to monitor such model systems in a high-throughput and high-content environment. Here, we demonstrate the ability to simultaneously track cell cycle phase and cell motion at the single cell level. Using a recently introduced cell cycle marker, we present a set of image analysis tools for automated cell phase analysis of live cells over extended time periods. Our model-based approach enables the characterization of the four phases of the cell cycle G1, S, G2, and M, which enables the study of the effect of inhibitor compounds that are designed to block the replication of cancerous cells in any of the phases. We approach the tracking problem as a spatio-temporal volume segmentation task, where the 2D slices are stacked into a volume with time as the z dimension. The segmentation of the G2 and S phases is accomplished using level sets, and we designed a model-based shape/size constraint to control the evolution of the level set. Our main contribution is the design of a speed function coupled with a fast marching path planning approach for tracking cells across the G1 phase based on the appearance change of the nuclei. The viability of our approach is demonstrated by presenting quantitative results on both controls and cases in which cells are treated with a cell cycle inhibitor.     

Three-dimensional label-free imaging and quantification of migrating cells during wound healing

The wound-healing assay is a simple but effective tool for studying collective cell migration (CCM) that is widely used in biophysical studies and high-throughput screening. However, conventional imaging and analysis methods only address two-dimensional (2D) properties in a wound healing assay, such as gap closure rate. This is unfortunate because biological cells are complex 3D structures, and their dynamics provide significant information about cell physiology. Here, we presented 3D label-free imaging for wound healing assays and investigated the 3D dynamics of CCM using optical diffraction tomography. High-resolution subcellular structures as well as their collective dynamics were imaged and analyzed quantitatively.     

基于正侧位锥束CT投影快速构建脊椎三维模型

目的 探讨基于C形臂的两张正侧位2D锥束CT（CBCT）投影图像进行3D模型重建的效果。方法 采用半自动化的二维投影图像特征点提取算法,选定18点的特征点集并提取其对应的正侧位二维投影图像平面坐标,然后针对C形臂CBCT系统建立坐标系,推导特征点三维空间坐标与其在投影图像中平面坐标之间的几何关系,代入转换公式获得特征点集的三维空间坐标。利用薄板样条法对三维脊椎基础模型进行空间非刚性插值获得三维脊椎目标模型。将L3石膏模型置入C形臂CBCT系统,获取375幅圆周扫描图像,利用FDK算法重建三维模型并进行表面重采样得到三维脊椎参考模型,对其进行不规则形状调制得到三维脊椎基础模型,利用本文方法对三维脊椎基础模型进行空间非刚性插值得到三维脊椎目标模型,并设置对照组对本文方法的精度进行评价。结果 相对于特征点手动提取和边缘增强提取算法,采用半自动化特征点提取算法构建获得的三维脊椎目标模型与参考模型的误差降低至1 mm以内。结论 采用本文方法可构建出近似的、精度较高的脊椎三维模型,为基于C形臂CBCT的手术导航提供3D图像支持。     

Deep pyramid local attention neural network for cardiac structure segmentation in two-dimensional echocardiography

Automatic semantic segmentation in 2D echocardiography is vital in clinical practice for assessing various cardiac functions and improving the diagnosis of cardiac diseases. However, two distinct problems have persisted in automatic segmentation in 2D echocardiography, namely the lack of an effective feature enhancement approach for contextual feature capture and lack of label coherence in category prediction for individual pixels. Therefore, in this study, we propose a deep learning model, called deep pyramid local attention neural network (PLANet), to improve the segmentation performance of automatic methods in 2D echocardiography. Specifically, we propose a pyramid local attention module to enhance features by capturing supporting information within compact and sparse neighboring contexts. We also propose a label coherence learning mechanism to promote prediction consistency for pixels and their neighbors by guiding the learning with explicit supervision signals. The proposed PLANet was extensively evaluated on the dataset of cardiac acquisitions for multi-structure ultrasound segmentation (CAMUS) and sub-EchoNet-Dynamic, which are two large-scale and public 2D echocardiography datasets. The experimental results show that PLANet performs better than traditional and deep learning-based segmentation methods on geometrical and clinical metrics. Moreover, PLANet can complete the segmentation of heart structures in 2D echocardiography in real time, indicating a potential to assist cardiologists accurately and efficiently.     

Image-based characterization of thrombus formation in time-lapse DIC microscopy

The characterization of thrombus formation in time-lapse DIC microscopy is of increased interest for identifying genes which account for atherothrombosis and coronary artery diseases (CADs). In particular, we are interested in large-scale studies on zebrafish, which result in large amount of data, and require automatic processing. In this work, we present an image-based solution for the automatized extraction of parameters quantifying the temporal development of thrombotic plugs. Our system is based on the joint segmentation of thrombotic and aortic regions over time. This task is made difficult by the low contrast and the high dynamic conditions observed in vivo DIC microscopic scenes. Our key idea is to perform this segmentation by distinguishing the different motion patterns in image time series rather than by solving standard image segmentation tasks in each image frame. Thus, we are able to compensate for the poor imaging conditions. We model motion patterns by energies based on the idea of dynamic textures, and regularize the model by two prior energies on the shape of the aortic region and on the topological relationship between the thrombus and the aorta. We demonstrate the performance of our segmentation algorithm by qualitative and quantitative experiments on synthetic examples as well as on real in vivo microscopic sequences.     

Interactive 3D editing tools for image segmentation

Segmentation is an important part of image processing, which often has a large impact on quantitative image analysis results. Fully automated operator independent segmentation procedures that successfully work in a population with a larger biological variation are extremely difficult to design and usually some kind of operator intervention is required, at least in pathological cases.We developed a variety of 3D editing tools that can be used to correct or improve results of initial automatic segmentation procedures. Specifically we will discuss and show examples for three types of editing tools that we termed: hole-filling (tool 1), point-bridging (tool 2), and surface-dragging (tool 3). Each tool comes in a number of flavors, all of which are implemented in a truly 3D manner. We describe the principles, evaluate efficiency and flexibility, and discuss advantages and disadvantages of each tool. We further demonstrate the superiority of the 3D approach over the time-consuming slice-by-slice editing of 3D datasets, which is still widely used in medical image processing today. We conclude that performance criteria for automatic segmentation algorithms may be eased significantly by including 3D editing tools early in the design process.     

A global benchmark of algorithms for segmenting the left atrium from late gadolinium-enhanced cardiac magnetic resonance imaging

Segmentation of medical images, particularly late gadolinium-enhanced magnetic resonance imaging (LGE-MRI) used for visualizing diseased atrial structures, is a crucial first step for ablation treatment of atrial fibrillation. However, direct segmentation of LGE-MRIs is challenging due to the varying intensities caused by contrast agents. Since most clinical studies have relied on manual, labor-intensive approaches, automatic methods are of high interest, particularly optimized machine learning approaches. To address this, we organized the 2018 Left Atrium Segmentation Challenge using 154 3D LGE-MRIs, currently the world's largest atrial LGE-MRI dataset, and associated labels of the left atrium segmented by three medical experts, ultimately attracting the participation of 27 international teams. In this paper, extensive analysis of the submitted algorithms using technical and biological metrics was performed by undergoing subgroup analysis and conducting hyper-parameter analysis, offering an overall picture of the major design choices of convolutional neural networks (CNNs) and practical considerations for achieving state-of-the-art left atrium segmentation. Results show that the top method achieved a Dice score of 93.2% and a mean surface to surface distance of 0.7 mm, significantly outperforming prior state-of-the-art. Particularly, our analysis demonstrated that double sequentially used CNNs, in which a first CNN is used for automatic region-of-interest localization and a subsequent CNN is used for refined regional segmentation, achieved superior results than traditional methods and machine learning approaches containing single CNNs. This large-scale benchmarking study makes a significant step towards much-improved segmentation methods for atrial LGE-MRIs, and will serve as an important benchmark for evaluating and comparing the future works in the field. Furthermore, the findings from this study can potentially be extended to other imaging datasets and modalities, having an impact on the wider medical imaging community.     

Computer-aided nidus segmentation and angiographic characterization of arteriovenous malformations

Purpose

Exact knowledge about the nidus of an arteriovenous malformation (AVM) and the connected vessels is often required for image-based research projects and optimal therapy planning. The aim of this work is to present and evaluate a computer-aided nidus segmentation technique and subsequent angiographic characterization of the connected vessels that can be visualized in 3D.

Methods

The proposed method was developed and evaluated based on 15 datasets of patients with an AVM. Each dataset consists of a high-resolution 3D and a 4D magnetic resonance angiography (MRA) image sequence. After automatic cerebrovascular segmentation from the 3D MRA dataset, a voxel-wise support vector machine classification based on four extracted features is performed to generate a new parameter map. The nidus is represented by positive values in this parameter map and can be extracted using volume growing. Finally, the nidus segmentation is dilated and used for an automatic identification of feeding arteries and draining veins by integrating hemodynamic information from the 4D MRA datasets.

Results

A quantitative comparison of the computer-aided AVM nidus segmentation results to manual gold-standard segmentations by two observers revealed a mean Dice coefficient of 0.835, which is comparable to the inter-observer agreement for which a mean Dice coefficient of 0.830 was determined. The angiographic characterization was visually rated feasible for all patients.

Conclusion

The presented computer-aided method enables a reproducible and fast extraction of the AVM nidus as well as an automatic angiographic characterization of the connected vessels, which can be used to support image-based research projects and therapy planning of AVMs.     

Rapid multi-plane phase-contrast microscopy reveals torsional dynamics in flagellar motion

High speed volumetric optical microscopy is an important tool for observing rapid processes in living cells or for real-time tracking of sub-cellular components. However, the 3D imaging capability often comes at the price of a high technical complexity of the imaging system and/or the requirement of demanding image analysis. Here, we propose a combination of conventional phase-contrast imaging with a customized multi-plane beam-splitter for enabling simultaneous acquisition of images in eight different focal planes. Our method is technically straightforward and does not require complex post-processing image analysis. We apply our multi-plane phase-contrast microscope to the real-time observation of the fast motion of reactivated Chlamydomonas axonemes with sub-µm spatial and 4 ms temporal resolution. Our system allows us to observe not only bending but also the three-dimensional torsional dynamics of these micro-swimmers.