3D image segmentation using combined shape-intensity prior models

Objective Statistical models for medical images have been developed to increase robustness in the segmentation process. In this project, a fully automatic approach to build a statistical shape-intensity model and combine this model with level set segmentation was designed, implemented and tested by applying the algorithm to clinical image data. Methods By using a hierarchical registration approach based on mutual information and demons registration, 3D statistical shape-intensity models were created by applying Principal Component Analysis. Using these models in combination with level set segmentation results in a fully automatic modeling and segmentation pipeline. Results Examples for shape-intensity models were synthesized and these models were used to automatically segment 3D MRI and CT images. Quantitative evaluation of the framework was performed by comparing automatic segmentation results to segmentation results of medical experts. Conclusion Evaluation tests in which this method was used for the automatic segmentation of femora and cardiac MRI endocardial surfaces are very promising. The implementation of an additional cost function term and the addition of information about the surroundings of an organ in the model are currently under development.     

3D image segmentation by using statistical deformation models and level sets

Purpose Improved segmentation of soft objects was sought using a new method that combines level set segmentation with statistical deformation models, using prior knowledge of the shape of an object as well as information derived from the input image. Methods Statistical deformation models were created using Euclidian distance functions of binary data and a multi-hierarchical registration approach based on mutual information metric and demons deformable registration. This approach is motivated by the fact that models based on signed distance maps, traditionally combined with level set segmentation can result in irregular shapes and do not establish explicit correspondences. By using statistical deformation models as representation of shape and a maximum a posteriori (MAP) estimation model to estimate the MAP shape of the object to be segmented, a robust segmentation algorithm using accurate shape models could be developed. Results The accuracy and correctness of the synthesized models was evaluated on different 3D objects (cardiac MRI and spinal CT vertebral segment) and the segmentation algorithm was validated by performing different segmentation tasks using various image modalities. The results of this evaluation are very promising and show the potential utility of the approach. Conclusion Initial results demonstrate the approach is feasible and may be advantageous over alternative segmentation methods. Extensions of the model, which also incorporate prior knowledge about the spatial distribution of grey values, are currently under development.     

4D deformable models with temporal constraints: application to 4D cardiac image segmentation

Segmentation of time series of 3D cardiac images is clinically used for the assessment of the mechanical function of the left ventricle. To take into account the 4D (3D+T) nature of those images, we propose to extend the deformable surface framework by introducing time-dependent constraints. Thus, in addition to computing an internal force for enforcing the regularity of the deformable model, prior motion knowledge is introduced in the deformation process through either temporal smoothing or trajectory constraints. In this paper, deformable surfaces are represented as simplex meshes owing to their generality and their ability to compute mean curvature at each vertex. The segmentation accuracy of this 4D deformable model is estimated on synthetic SPECT image sequences for which a ground truth about the LV volume is known. Segmentation of non-synthetic SPECT and other modalities 4D images is also discussed.     

Atlas-based segmentation of 3D cerebral structures with competitive level sets and fuzzy control

We propose a novel approach for the simultaneous segmentation of multiple structures with competitive level sets driven by fuzzy control. To this end, several contours evolve simultaneously toward previously defined anatomical targets. A fuzzy decision system combines the a priori knowledge provided by an anatomical atlas with the intensity distribution of the image and the relative position of the contours. This combination automatically determines the directional term of the evolution equation of each level set. This leads to a local expansion or contraction of the contours, in order to match the boundaries of their respective targets. Two applications are presented: the segmentation of the brain hemispheres and the cerebellum, and the segmentation of deep internal structures. Experimental results on real magnetic resonance (MR) images are presented, quantitatively assessed and discussed.     

Segmentation and quantification of the aortic arch using joint 3D model-based segmentation and elastic image registration

Accurate quantification of the morphology of vessels is important for diagnosis and treatment of cardiovascular diseases. We introduce a new joint segmentation and registration approach for the quantification of the aortic arch morphology that combines 3D model-based segmentation with elastic image registration. With this combination, the approach benefits from the robustness of model-based segmentation and the accuracy of elastic registration. The approach can cope with a large spectrum of vessel shapes and particularly with pathological shapes that deviate significantly from the underlying model used for segmentation. The performance of the approach has been evaluated on the basis of 3D synthetic images, 3D phantom data, and clinical 3D CTA images including pathologies. We also performed a quantitative comparison with previous approaches.     

Automatic 3D segmentation of individual facial muscles using unlabeled prior information

Purpose  

Segmentation of facial soft tissues is required for surgical planning and evaluation, but this is laborious using manual methods and has been difficult to achieve with digital segmentation methods. A new automatic 3D segmentation method for facial soft tissues in magnetic resonance imaging (MRI) images was designed, implemented, and tested.     

Statistical shape models for 3D medical image segmentation: A review

Statistical shape models (SSMs) have by now been firmly established as a robust tool for segmentation of medical images. While 2D models have been in use since the early 1990s, wide-spread utilization of three-dimensional models appeared only in recent years, primarily made possible by breakthroughs in automatic detection of shape correspondences. In this article, we review the techniques required to create and employ these 3D SSMs. While we concentrate on landmark-based shape representations and thoroughly examine the most popular variants of Active Shape and Active Appearance models, we also describe several alternative approaches to statistical shape modeling. Structured into the topics of shape representation, model construction, shape correspondence, local appearance models and search algorithms, we present an overview of the current state of the art in the field. We conclude with a survey of applications in the medical field and a discussion of future developments.     

Fully automatic segmentation of the mitral leaflets in 3D transesophageal echocardiographic images using multi-atlas joint label fusion and deformable medial modeling

Comprehensive visual and quantitative analysis of in vivo human mitral valve morphology is central to the diagnosis and surgical treatment of mitral valve disease. Real-time 3D transesophageal echocardiography (3D TEE) is a practical, highly informative imaging modality for examining the mitral valve in a clinical setting. To facilitate visual and quantitative 3D TEE image analysis, we describe a fully automated method for segmenting the mitral leaflets in 3D TEE image data. The algorithm integrates complementary probabilistic segmentation and shape modeling techniques (multi-atlas joint label fusion and deformable modeling with continuous medial representation) to automatically generate 3D geometric models of the mitral leaflets from 3D TEE image data. These models are unique in that they establish a shape-based coordinate system on the valves of different subjects and represent the leaflets volumetrically, as structures with locally varying thickness. In this work, expert image analysis is the gold standard for evaluating automatic segmentation. Without any user interaction, we demonstrate that the automatic segmentation method accurately captures patient-specific leaflet geometry at both systole and diastole in 3D TEE data acquired from a mixed population of subjects with normal valve morphology and mitral valve disease.     

Anatomically constrained and attention-guided deep feature fusion for joint segmentation and deformable medical image registration

The main objective of anatomically plausible results for deformable image registration is to improve model’s registration accuracy by minimizing the difference between a pair of fixed and moving images. Since many anatomical features are closely related to each other, leveraging supervision from auxiliary tasks (such as supervised anatomical segmentation) has the potential to enhance the realism of the warped images after registration. In this work, we employ a Multi-Task Learning framework to formulate registration and segmentation as a joint issue, in which we utilize anatomical constraint from auxiliary supervised segmentation to enhance the realism of the predicted images. First, we propose a Cross-Task Attention Block to fuse the high-level feature from both the registration and segmentation network. With the help of initial anatomical segmentation, the registration network can benefit from learning the task-shared feature correlation and rapidly focusing on the parts that need deformation. On the other hand, the anatomical segmentation discrepancy from ground-truth fixed annotations and predicted segmentation maps of initial warped images are integrated into the loss function to guide the convergence of the registration network. Ideally, a good deformation field should be able to minimize the loss function of registration and segmentation. The voxel-wise anatomical constraint inferred from segmentation helps the registration network to reach a global optimum for both deformable and segmentation learning. Both networks can be employed independently during the testing phase, enabling only the registration output to be predicted when the segmentation labels are unavailable. Qualitative and quantitative results indicate that our proposed methodology significantly outperforms the previous state-of-the-art approaches on inter-patient brain MRI registration and pre- and intra-operative uterus MRI registration tasks within our specific experimental setup, which leads to state-of-the-art registration quality scores of 0.755 and 0.731 (i.e., by 0.8% and 0.5% increases) DSC for both tasks, respectively.     

Segmentation of brain 3D MR images using level sets and dense registration

This paper presents a strategy for the segmentation of brain from volumetric MR images which integrates 3D segmentation and 3D registration processes. The segmentation process is based on the level set formalism. A closed 3D surface propagates towards the desired boundaries through the iterative evolution of a 4D implicit function. In this work, the propagation relies on a robust evolution model including adaptive parameters. These depend on the input data and on statistical distribution models. The main contribution of this paper is the use of an automatic registration method to initialize the surface, as an alternative solution to manual initialization. The registration is achieved through a robust multiresolution and multigrid minimization scheme. This coupling significantly improves the quality of the method, since the segmentation is faster, more reliable and fully automatic. Quantitative and qualitative results on both synthetic and real volumetric brain MR images are presented and discussed.     

Interactive 3D editing tools for image segmentation

Segmentation is an important part of image processing, which often has a large impact on quantitative image analysis results. Fully automated operator independent segmentation procedures that successfully work in a population with a larger biological variation are extremely difficult to design and usually some kind of operator intervention is required, at least in pathological cases.We developed a variety of 3D editing tools that can be used to correct or improve results of initial automatic segmentation procedures. Specifically we will discuss and show examples for three types of editing tools that we termed: hole-filling (tool 1), point-bridging (tool 2), and surface-dragging (tool 3). Each tool comes in a number of flavors, all of which are implemented in a truly 3D manner. We describe the principles, evaluate efficiency and flexibility, and discuss advantages and disadvantages of each tool. We further demonstrate the superiority of the 3D approach over the time-consuming slice-by-slice editing of 3D datasets, which is still widely used in medical image processing today. We conclude that performance criteria for automatic segmentation algorithms may be eased significantly by including 3D editing tools early in the design process.     

DeU-Net 2.0: Enhanced deformable U-Net for 3D cardiac cine MRI segmentation

Automatic segmentation of cardiac magnetic resonance imaging (MRI) facilitates efficient and accurate volume measurement in clinical applications. However, due to anisotropic resolution, ambiguous borders and complicated shapes, existing methods suffer from the degradation of accuracy and robustness in cardiac MRI segmentation. In this paper, we propose an enhanced Deformable U-Net (DeU-Net) for 3D cardiac cine MRI segmentation, composed of three modules, namely Temporal Deformable Aggregation Module (TDAM), Enhanced Deformable Attention Network (EDAN), and Probabilistic Noise Correction Module (PNCM). TDAM first takes consecutive cardiac MR slices (including a target slice and its neighboring reference slices) as input, and extracts spatio-temporal information by an offset prediction network to generate fused features of the target slice. Then the fused features are also fed into EDAN that exploits several flexible deformable convolutional layers and generates clear borders of every segmentation map. A Multi-Scale Attention Module (MSAM) in EDAN is proposed to capture long range dependencies between features of different scales. Meanwhile, PNCM treats the fused features as a distribution to quantify uncertainty. Experimental results show that our DeU-Net achieves the state-of-the-art performance in terms of the commonly used evaluation metrics on the Extended ACDC dataset and competitive performance on other two datasets, validating the robustness and generalization of DeU-Net.     

A hybrid framework for 3D medical image segmentation

In this paper we propose a novel hybrid 3D segmentation framework which combines Gibbs models, marching cubes and deformable models. In the framework, first we construct a new Gibbs model whose energy function is defined on a high order clique system. The new model includes both region and boundary information during segmentation. Next we improve the original marching cubes method to construct 3D meshes from Gibbs models' output. The 3D mesh serves as the initial geometry of the deformable model. Then we deform the deformable model using external image forces so that the model converges to the object surface. We run the Gibbs model and the deformable model recursively by updating the Gibbs model's parameters using the region and boundary information in the deformable model segmentation result. In our approach, the hybrid combination of region-based methods and boundary-based methods results in improved segmentations of complex structures. The benefit of the methodology is that it produces high quality segmentations of 3D structures using little prior information and minimal user intervention. The modules in this segmentation methodology are developed within the context of the Insight ToolKit (ITK). We present experimental segmentation results of brain tumors and evaluate our method by comparing experimental results with expert manual segmentations. The evaluation results show that the methodology achieves high quality segmentation results with computational efficiency. We also present segmentation results of other clinical objects to illustrate the strength of the methodology as a generic segmentation framework.     

CAN3D: Fast 3D medical image segmentation via compact context aggregation

Direct automatic segmentation of objects in 3D medical imaging, such as magnetic resonance (MR) imaging, is challenging as it often involves accurately identifying multiple individual structures with complex geometries within a large volume under investigation. Most deep learning approaches address these challenges by enhancing their learning capability through a substantial increase in trainable parameters within their models. An increased model complexity will incur high computational costs and large memory requirements unsuitable for real-time implementation on standard clinical workstations, as clinical imaging systems typically have low-end computer hardware with limited memory and CPU resources only. This paper presents a compact convolutional neural network (CAN3D) designed specifically for clinical workstations and allows the segmentation of large 3D Magnetic Resonance (MR) images in real-time. The proposed CAN3D has a shallow memory footprint to reduce the number of model parameters and computer memory required for state-of-the-art performance and maintain data integrity by directly processing large full-size 3D image input volumes with no patches required. The proposed architecture significantly reduces computational costs, especially for inference using the CPU. We also develop a novel loss function with extra shape constraints to improve segmentation accuracy for imbalanced classes in 3D MR images. Compared to state-of-the-art approaches (U-Net3D, improved U-Net3D and V-Net), CAN3D reduced the number of parameters up to two orders of magnitude and achieved much faster inference, up to 5 times when predicting with a standard commercial CPU (instead of GPU). For the open-access OAI-ZIB knee MR dataset, in comparison with manual segmentation, CAN3D achieved Dice coefficient values of (mean = 0.87 ± 0.02 and 0.85 ± 0.04) with mean surface distance errors (mean = 0.36 ± 0.32 mm and 0.29 ± 0.10 mm) for imbalanced classes such as (femoral and tibial) cartilage volumes respectively when training volume-wise under only 12G video memory. Similarly, CAN3D demonstrated high accuracy and efficiency on a pelvis 3D MR imaging dataset for prostate cancer consisting of 211 examinations with expert manual semantic labels (bladder, body, bone, rectum, prostate) now released publicly for scientific use as part of this work.     

Automatic abdominal multi-organ segmentation using deep convolutional neural network and time-implicit level sets

Purpose

Multi-organ segmentation from CT images is an essential step for computer-aided diagnosis and surgery planning. However, manual delineation of the organs by radiologists is tedious, time-consuming and poorly reproducible. Therefore, we propose a fully automatic method for the segmentation of multiple organs from three-dimensional abdominal CT images.

Methods

The proposed method employs deep fully convolutional neural networks (CNNs) for organ detection and segmentation, which is further refined by a time-implicit multi-phase evolution method. Firstly, a 3D CNN is trained to automatically localize and delineate the organs of interest with a probability prediction map. The learned probability map provides both subject-specific spatial priors and initialization for subsequent fine segmentation. Then, for the refinement of the multi-organ segmentation, image intensity models, probability priors as well as a disjoint region constraint are incorporated into an unified energy functional. Finally, a novel time-implicit multi-phase level-set algorithm is utilized to efficiently optimize the proposed energy functional model.

Results

Our method has been evaluated on 140 abdominal CT scans for the segmentation of four organs (liver, spleen and both kidneys). With respect to the ground truth, average Dice overlap ratios for the liver, spleen and both kidneys are 96.0, 94.2 and 95.4%, respectively, and average symmetric surface distance is less than 1.3 mm for all the segmented organs. The computation time for a CT volume is 125 s in average. The achieved accuracy compares well to state-of-the-art methods with much higher efficiency.

Conclusion

A fully automatic method for multi-organ segmentation from abdominal CT images was developed and evaluated. The results demonstrated its potential in clinical usage with high effectiveness, robustness and efficiency.

     

Unsupervised 4D myocardium segmentation with a Markov Random Field based deformable model

A stochastic deformable model is proposed for the segmentation of the myocardium in Magnetic Resonance Imaging. The segmentation is posed as a probabilistic optimization problem in which the optimal time-dependent surface is obtained for the myocardium of the heart in a discrete space of locations built upon simple geometric assumptions. For this purpose, first, the left ventricle is detected by a set of image analysis tools gathered from the literature. Then, the segmentation solution is obtained by the Maximization of the Posterior Marginals for the myocardium location in a Markov Random Field framework which optimally integrates temporal-spatial smoothness with intensity and gradient related features in an unsupervised way by the Maximum Likelihood estimation of the parameters of the field. This scheme provides a flexible and robust segmentation method which has been able to generate results comparable to manually segmented images for some derived cardiac function parameters in a set of 43 patients affected in different degrees by an Acute Myocardial Infarction.     

MRT of the knee joint cartilage using 3D volume imaging in combination with a fast image processor

From the knees of healthy persons and cadavers 2D-spin-echo-sequences were compared to 3D-gradient-echo-sequences (FLASH, FISP). We were able to show that 3D-volume-imaging, combined with image processing on a fast computer, is superior to 2D-spin-echo-imaging. The advantages are: slice thickness lower than 1 mm, secondary reconstruction in any desired plane, good contrast between intraarticular fluid, the meniscus and the hyaline cartilage. For the practical application of 3D-volume-imaging resulting in multiple images the secondary reconstruction and post processing of the large numbers of images with an image processing computer is necessary.     

Statistical deformable bone models for robust 3D surface extrapolation from sparse data

A majority of pre-operative planning and navigational guidance during computer assisted orthopaedic surgery routinely uses three-dimensional models of patient anatomy. These models enhance the surgeon's capability to decrease the invasiveness of surgical procedures and increase their accuracy and safety. A common approach for this is to use computed tomography (CT) or magnetic resonance imaging (MRI). These have the disadvantages that they are expensive and/or induce radiation to the patient. In this paper we propose a novel method to construct a patient-specific three-dimensional model that provides an appropriate intra-operative visualization without the need for a pre or intra-operative imaging. The 3D model is reconstructed by fitting a statistical deformable model to minimal sparse 3D data consisting of digitized landmarks and surface points that are obtained intra-operatively. The statistical model is constructed using Principal Component Analysis from training objects. Our deformation scheme efficiently and accurately computes a Mahalanobis distance weighted least square fit of the deformable model to the 3D data. Relaxing the Mahalanobis distance term as additional points are incorporated enables our method to handle small and large sets of digitized points efficiently. Formalizing the problem as a linear equation system helps us to provide real-time updates to the surgeons. Incorporation of M-estimator based weighting of the digitized points enables us to effectively reject outliers and compute stable models. We present here our evaluation results using leave-one-out experiments and extended validation of our method on nine dry cadaver bones.     

Automatic multimodal 2D/3D breast image registration using biomechanical FEM models and intensity-based optimization

Due to their different physical origin, X-ray mammography and Magnetic Resonance Imaging (MRI) provide complementary diagnostic information. However, the correlation of their images is challenging due to differences in dimensionality, patient positioning and compression state of the breast. Our automated registration takes over part of the correlation task. The registration method is based on a biomechanical finite element model, which is used to simulate mammographic compression. The deformed MRI volume can be compared directly with the corresponding mammogram. The registration accuracy is determined by a number of patient-specific parameters. We optimize these parameters – e.g. breast rotation – using image similarity measures. The method was evaluated on 79 datasets from clinical routine. The mean target registration error was 13.2 mm in a fully automated setting. On basis of our results, we conclude that a completely automated registration of volume images with 2D mammograms is feasible. The registration accuracy is within the clinically relevant range and thus beneficial for multimodal diagnosis.