A hybrid framework for 3D medical image segmentation

In this paper we propose a novel hybrid 3D segmentation framework which combines Gibbs models, marching cubes and deformable models. In the framework, first we construct a new Gibbs model whose energy function is defined on a high order clique system. The new model includes both region and boundary information during segmentation. Next we improve the original marching cubes method to construct 3D meshes from Gibbs models' output. The 3D mesh serves as the initial geometry of the deformable model. Then we deform the deformable model using external image forces so that the model converges to the object surface. We run the Gibbs model and the deformable model recursively by updating the Gibbs model's parameters using the region and boundary information in the deformable model segmentation result. In our approach, the hybrid combination of region-based methods and boundary-based methods results in improved segmentations of complex structures. The benefit of the methodology is that it produces high quality segmentations of 3D structures using little prior information and minimal user intervention. The modules in this segmentation methodology are developed within the context of the Insight ToolKit (ITK). We present experimental segmentation results of brain tumors and evaluate our method by comparing experimental results with expert manual segmentations. The evaluation results show that the methodology achieves high quality segmentation results with computational efficiency. We also present segmentation results of other clinical objects to illustrate the strength of the methodology as a generic segmentation framework.     

Segmentation of the skull in MRI volumes using deformable model and taking the partial volume effect into account

Segmentation of the skull in medical imagery is an important stage in applications that require the construction of realistic models of the head. Such models are used, for example, to simulate the behavior of electro-magnetic fields in the head and to model the electrical activity of the cortex in EEG and MEG data. In this paper, we present a new approach for segmenting regions of bone in MRI volumes using deformable models. Our method takes into account the partial volume effects that occur with MRI data, thus permitting a precise segmentation of these bone regions. At each iteration of the propagation of the model, partial volume is estimated in a narrow band around the deformable model. Our segmentation method begins with a pre-segmentation stage, in which a preliminary segmentation of the skull is constructed using a region-growing method. The surface that bounds the pre-segmented skull region offers an automatic 3D initialization of the deformable model. This surface is then propagated (in 3D) in the direction of its normal. This propagation is achieved using level set method, thus permitting changes to occur in the topology of the surface as it evolves, an essential capability for our problem. The speed at which the surface evolves is a function of the estimated partial volume. This provides a sub-voxel accuracy in the resulting segmentation.     

3D image segmentation by using statistical deformation models and level sets

Purpose Improved segmentation of soft objects was sought using a new method that combines level set segmentation with statistical deformation models, using prior knowledge of the shape of an object as well as information derived from the input image. Methods Statistical deformation models were created using Euclidian distance functions of binary data and a multi-hierarchical registration approach based on mutual information metric and demons deformable registration. This approach is motivated by the fact that models based on signed distance maps, traditionally combined with level set segmentation can result in irregular shapes and do not establish explicit correspondences. By using statistical deformation models as representation of shape and a maximum a posteriori (MAP) estimation model to estimate the MAP shape of the object to be segmented, a robust segmentation algorithm using accurate shape models could be developed. Results The accuracy and correctness of the synthesized models was evaluated on different 3D objects (cardiac MRI and spinal CT vertebral segment) and the segmentation algorithm was validated by performing different segmentation tasks using various image modalities. The results of this evaluation are very promising and show the potential utility of the approach. Conclusion Initial results demonstrate the approach is feasible and may be advantageous over alternative segmentation methods. Extensions of the model, which also incorporate prior knowledge about the spatial distribution of grey values, are currently under development.     

Efficient and generalizable statistical models of shape and appearance for analysis of cardiac MRI

We present a framework for the analysis of short axis cardiac MRI, using statistical models of shape and appearance. The framework integrates temporal and structural constraints and avoids common optimization problems inherent in such high dimensional models. The first contribution is the introduction of an algorithm for fitting 3D active appearance models (AAMs) on short axis cardiac MRI. We observe a 44-fold increase in fitting speed and a segmentation accuracy that is on par with Gauss-Newton optimization, one of the most widely used optimization algorithms for such problems. The second contribution involves an investigation on hierarchical 2D+time active shape models (ASMs), that integrate temporal constraints and simultaneously improve the 3D AAM based segmentation. We obtain encouraging results (endocardial/epicardial error 1.43+/-0.49 mm/1.51+/-0.48 mm) on 7980 short axis cardiac MR images acquired from 33 subjects. We have placed our dataset online, for the community to use and build upon.     

4D deformable models with temporal constraints: application to 4D cardiac image segmentation

Segmentation of time series of 3D cardiac images is clinically used for the assessment of the mechanical function of the left ventricle. To take into account the 4D (3D+T) nature of those images, we propose to extend the deformable surface framework by introducing time-dependent constraints. Thus, in addition to computing an internal force for enforcing the regularity of the deformable model, prior motion knowledge is introduced in the deformation process through either temporal smoothing or trajectory constraints. In this paper, deformable surfaces are represented as simplex meshes owing to their generality and their ability to compute mean curvature at each vertex. The segmentation accuracy of this 4D deformable model is estimated on synthetic SPECT image sequences for which a ground truth about the LV volume is known. Segmentation of non-synthetic SPECT and other modalities 4D images is also discussed.     

3D variational brain tumor segmentation using Dirichlet priors on a clustered feature set

Purpose

Brain tumor segmentation is a required step before any radiation treatment or surgery. When performed manually, segmentation is time consuming and prone to human errors. Therefore, there have been significant efforts to automate the process. But, automatic tumor segmentation from MRI data is a particularly challenging task. Tumors have a large diversity in shape and appearance with intensities overlapping the normal brain tissues. In addition, an expanding tumor can also deflect and deform nearby tissue. In our work, we propose an automatic brain tumor segmentation method that addresses these last two difficult problems.

Methods

We use the available MRI modalities (T1, T1c, T2) and their texture characteristics to construct a multidimensional feature set. Then, we extract clusters which provide a compact representation of the essential information in these features. The main idea in this work is to incorporate these clustered features into the 3D variational segmentation framework. In contrast to previous variational approaches, we propose a segmentation method that evolves the contour in a supervised fashion. The segmentation boundary is driven by the learned region statistics in the cluster space. We incorporate prior knowledge about the normal brain tissue appearance during the estimation of these region statistics. In particular, we use a Dirichlet prior that discourages the clusters from the normal brain region to be in the tumor region. This leads to a better disambiguation of the tumor from brain tissue.

Results

We evaluated the performance of our automatic segmentation method on 15 real MRI scans of brain tumor patients, with tumors that are inhomogeneous in appearance, small in size and in proximity to the major structures in the brain. Validation with the expert segmentation labels yielded encouraging results: Jaccard (58%), Precision (81%), Recall (67%), Hausdorff distance (24 mm).

Conclusions

Using priors on the brain/tumor appearance, our proposed automatic 3D variational segmentation method was able to better disambiguate the tumor from the surrounding tissue.

     

Deformable segmentation via sparse representation and dictionary learning

“Shape” and “appearance”, the two pillars of a deformable model, complement each other in object segmentation. In many medical imaging applications, while the low-level appearance information is weak or mis-leading, shape priors play a more important role to guide a correct segmentation, thanks to the strong shape characteristics of biological structures. Recently a novel shape prior modeling method has been proposed based on sparse learning theory. Instead of learning a generative shape model, shape priors are incorporated on-the-fly through the sparse shape composition (SSC). SSC is robust to non-Gaussian errors and still preserves individual shape characteristics even when such characteristics is not statistically significant.Although it seems straightforward to incorporate SSC into a deformable segmentation framework as shape priors, the large-scale sparse optimization of SSC has low runtime efficiency, which cannot satisfy clinical requirements. In this paper, we design two strategies to decrease the computational complexity of SSC, making a robust, accurate and efficient deformable segmentation system. (1) When the shape repository contains a large number of instances, which is often the case in 2D problems, K-SVD is used to learn a more compact but still informative shape dictionary. (2) If the derived shape instance has a large number of vertices, which often appears in 3D problems, an affinity propagation method is used to partition the surface into small sub-regions, on which the sparse shape composition is performed locally. Both strategies dramatically decrease the scale of the sparse optimization problem and hence speed up the algorithm. Our method is applied on a diverse set of biomedical image analysis problems. Compared to the original SSC, these two newly-proposed modules not only significant reduce the computational complexity, but also improve the overall accuracy.     

Segmentation by retrieval with guided random walks: Application to left ventricle segmentation in MRI

In this paper, a new segmentation framework with prior knowledge is proposed and applied to the left ventricles in cardiac Cine MRI sequences. We introduce a new formulation of the random walks method, coined as guided random walks, in which prior knowledge is integrated seamlessly. In comparison with existing approaches that incorporate statistical shape models, our method does not extract any principal model of the shape or appearance of the left ventricle. Instead, segmentation is accompanied by retrieving the closest subject in the database that guides the segmentation the best. Using this techniques, rare cases can also effectively exploit prior knowledge from few samples in training set. These cases are usually disregarded in statistical shape models as they are outnumbered by frequent cases (effect of class population). In the worst-case scenario, if there is no matching case in the database to guide the segmentation, performance of the proposed method reaches to the conventional random walks, which is shown to be accurate if sufficient number of seeds is provided. There is a fast solution to the proposed guided random walks by using sparse linear matrix operations and the whole framework can be seamlessly implemented in a parallel architecture. The method has been validated on a comprehensive clinical dataset of 3D+t short axis MR images of 104 subjects from 5 categories (normal, dilated left ventricle, ventricular hypertrophy, recent myocardial infarction, and heart failure). The average segmentation errors were found to be 1.54 mm for the endocardium and 1.48 mm for the epicardium. The method was validated by measuring different algorithmic and physiologic indices and quantified with manual segmentation ground truths, provided by a cardiologist.     

Myocardial deformation recovery from cine MRI using a nearly incompressible biventricular model

This paper presents a method for biventricular myocardial deformation recovery from cine MRI. The method is based on a deformable model that is nearly incompressible, a desirable property since the myocardium has been shown to be nearly incompressible. The model uses a matrix-valued radial basis function to represent divergence-free displacement fields, which is a first order approximation of incompressibility. This representation allows for deformation modeling of an arbitrary topologies with a relatively small number of parameters, which is suitable for representing the motion of the multi-chamber structure of the heart. The myocardium needs to be segmented in an initial frame after which the method automatically determines the tissue deformation everywhere in the myocardium throughout the cardiac cycle. Two studies were carried out to validate the method. In the first study the myocardial deformation was recovered from a 3D anatomical cine MRI sequence of a healthy volunteer and then validated against the manual segmentation of the biventricular wall and against the corresponding 3D tagged cine MRI sequence. The average volume agreement between the model and the manual segmentation had a false positive rate of 3.2%, false negative rate of 2.8% and true positive rate of 91.4%. The average distance between the model and manually determined intersections of perpendicular tag planes was 1.7mm (1.2 pixel). The same procedures was repeated on another set of 3D anatomical and tagged MRI scans of the same volunteer taken four months later. The recovered deformation was very similar to the one obtained from the first set of scans. In the second study the method was applied to 3D anatomical cine MRI scans of three patients with ventricular dyssynchrony and three age-matched healthy volunteers. The recovered strains of the normal subjects were clearly stronger than the recovered strains of the patients and they were similar to those reported by other researchers. The recovered deformation of all six subjects was validated against manual segmentation of the biventricular wall and against corresponding tagged MRI scans. The agreement was similar to that of the first study.     

Non-parametric geodesic active regions: method and evaluation for cerebral aneurysms segmentation in 3DRA and CTA

Segmentation of vascular structures is a difficult and challenging task. In this article, we present an algorithm devised for the segmentation of such structures. Our technique consists in a geometric deformable model with associated energy functional that incorporates high-order multiscale features in a non-parametric statistical framework. Although the proposed segmentation method is generic, it has been applied to the segmentation of cerebral aneurysms in 3DRA and CTA. An evaluation study over 10 clinical datasets indicate that the segmentations obtained by our method present a high overlap index with respect to the ground-truth (91.13% and 73.31%, respectively) and that the mean error distance from the surface to the ground truth is close to the in-plane resolution (0.40 and 0.38 mm, respectively). Besides, our technique favorably compares to other alternative techniques based on deformable models, namely parametric geodesic active regions and active contours without edges.     

A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac MRI

Segmentation of the left ventricle (LV) from cardiac magnetic resonance imaging (MRI) datasets is an essential step for calculation of clinical indices such as ventricular volume and ejection fraction. In this work, we employ deep learning algorithms combined with deformable models to develop and evaluate a fully automatic LV segmentation tool from short-axis cardiac MRI datasets. The method employs deep learning algorithms to learn the segmentation task from the ground true data. Convolutional networks are employed to automatically detect the LV chamber in MRI dataset. Stacked autoencoders are used to infer the LV shape. The inferred shape is incorporated into deformable models to improve the accuracy and robustness of the segmentation. We validated our method using 45 cardiac MR datasets from the MICCAI 2009 LV segmentation challenge and showed that it outperforms the state-of-the art methods. Excellent agreement with the ground truth was achieved. Validation metrics, percentage of good contours, Dice metric, average perpendicular distance and conformity, were computed as 96.69%, 0.94, 1.81 mm and 0.86, versus those of $79.2-95.62\%,$ 0.87–0.9, 1.76–2.97 mm and 0.67–0.78, obtained by other methods, respectively.     

Joint level-set shape modeling and appearance modeling for brain structure segmentation

This paper presents a new fully automatic model-based segmentation algorithm, which combines level-set methods to model the shape of brain structures and their variation with active appearance modeling to generate images that are used to drive the segmentation. The new algorithm incorporates multi-modality images to improve the segmentation performance and the recursive least square (RLS) algorithm is adopted to minimize the difference between test image and the one synthesized from the shape and appearance modeling. When compared with manual segmentation, the 2D and 3D experiments demonstrate that the new algorithm is computationally efficient and robust and is promising for automatic segmentation of the lateral ventricles.     

CAN: Context-assisted full Attention Network for brain tissue segmentation

Brain tissue segmentation is of great value in diagnosing brain disorders. Three-dimensional (3D) and two-dimensional (2D) segmentation methods for brain Magnetic Resonance Imaging (MRI) suffer from high time complexity and low segmentation accuracy, respectively. To address these two issues, we propose a Context-assisted full Attention Network (CAN) for brain MRI segmentation by integrating 2D and 3D data of MRI. Different from the fully symmetric structure U-Net, the CAN takes the current 2D slice, its 3D contextual skull slices and 3D contextual brain slices as the input, which are further encoded by the DenseNet and decoded by our constructed full attention network. We have validated the effectiveness of the CAN on our collected dataset PWML and two public datasets dHCP2017 and MALC2012. Our code is available at https://github.com/nwuAI/CAN.     

Active appearance pyramids for object parametrisation and fitting

Object class representation is one of the key problems in various medical image analysis tasks. We propose a part-based parametric appearance model we refer to as an Active Appearance Pyramid (AAP). The parts are delineated by multi-scale Local Feature Pyramids (LFPs) for superior spatial specificity and distinctiveness. An AAP models the variability within a population with local translations of multi-scale parts and linear appearance variations of the assembly of the parts. It can fit and represent new instances by adjusting the shape and appearance parameters. The fitting process uses a two-step iterative strategy: local landmark searching followed by shape regularisation. We present a simultaneous local feature searching and appearance fitting algorithm based on the weighted Lucas and Kanade method. A shape regulariser is derived to calculate the maximum likelihood shape with respect to the prior and multiple landmark candidates from multi-scale LFPs, with a compact closed-form solution. We apply the 2D AAP on the modelling of variability in patients with lumbar spinal stenosis (LSS) and validate its performance on 200 studies consisting of routine axial and sagittal MRI scans. Intervertebral sagittal and parasagittal cross-sections are typically used for the diagnosis of LSS, we therefore build three AAPs on L3/4, L4/5 and L5/S1 axial cross-sections and three on parasagittal slices. Experiments show significant improvement in convergence range, robustness to local minima and segmentation precision compared with Constrained Local Models (CLMs), Active Shape Models (ASMs) and Active Appearance Models (AAMs), as well as superior performance in appearance reconstruction compared with AAMs. We also validate the performance on 3D CT volumes of hip joints from 38 studies. Compared to AAMs, AAPs achieve a higher segmentation and reconstruction precision. Moreover, AAPs have a significant improvement in efficiency, consuming about half the memory and less than 10% of the training time and 15% of the testing time.     

Shape-based cortical surface segmentation for visualization brain mapping

We describe a knowledge-based approach to cortical surface segmentation that uses learned knowledge of the overall shape and range of variation of the cortex (excluding the detailed gyri and sulci) to guide the search for the grey-CSF boundary in a structural MRI image volume. The shape knowledge is represented by a radial surface model, which is a type of geometric constraint network (GCN) that we hypothesize can represent shape by networks of locally interacting constraints. The shape model is used in a protocol for visualization-based mapping of cortical stimulation mapping (CSM) sites onto the brain surface, prior to integration with other mapping modalities or as input to existing surface analysis and reconfiguration programs. Example results are presented for CSM data related to language organization in the cortex, but the methods should be applicable to other situations where a realistic visualization of the brain surface, as seen at neurosurgery, is desired.     

Deformable appearance pyramids for anatomy representation,landmark detection and pathology classification

Purpose

Representation of anatomy appearance is one of the key problems in medical image analysis. An appearance model represents the anatomies with parametric forms, which are then vectorised for prior learning, segmentation and classification tasks.

Methods

We propose a part-based parametric appearance model we refer to as a deformable appearance pyramid (DAP). The parts are delineated by multi-scale local feature pyramids extracted from an image pyramid. Each anatomy is represented by an appearance pyramid, with the variability within a population approximated by local translations of the multi-scale parts and linear appearance variations in the assembly of the parts. We introduce DAPs built on two types of image pyramids, namely Gaussian and wavelet pyramids, and present two approaches to model the prior and fit the model, one explicitly using a subspace Lucas–Kanade algorithm and the other implicitly using the supervised descent method (SDM).

Results

We validate the performance of the DAP instances with difference configurations on the problem of lumbar spinal stenosis for localising the landmarks and classifying the pathologies. We also compare them with classic methods such as active shape models, active appearance models and constrained local models. Experimental results show that the DAP built on wavelet pyramids and fitted with SDM gives the best results in both landmark localisation and classification.

Conclusion

A new appearance model is introduced with several configurations presented and evaluated. The DAPs can be readily applied for other clinical problems for the tasks of prior learning, landmark detection and pathology classification.

     

CEREBRUM: a fast and fully-volumetric Convolutional Encoder-decodeR for weakly-supervised sEgmentation of BRain strUctures from out-of-the-scanner MRI

Many functional and structural neuroimaging studies call for accurate morphometric segmentation of different brain structures starting from image intensity values of MRI scans. Current automatic (multi-) atlas-based segmentation strategies often lack accuracy on difficult-to-segment brain structures and, since these methods rely on atlas-to-scan alignment, they may take long processing times. Alternatively, recent methods deploying solutions based on Convolutional Neural Networks (CNNs) are enabling the direct analysis of out-of-the-scanner data. However, current CNN-based solutions partition the test volume into 2D or 3D patches, which are processed independently. This process entails a loss of global contextual information, thereby negatively impacting the segmentation accuracy. In this work, we design and test an optimised end-to-end CNN architecture that makes the exploitation of global spatial information computationally tractable, allowing to process a whole MRI volume at once. We adopt a weakly supervised learning strategy by exploiting a large dataset composed of 947 out-of-the-scanner (3 Tesla T1-weighted 1mm isotropic MP-RAGE 3D sequences) MR Images. The resulting model is able to produce accurate multi-structure segmentation results in only a few seconds. Different quantitative measures demonstrate an improved accuracy of our solution when compared to state-of-the-art techniques. Moreover, through a randomised survey involving expert neuroscientists, we show that subjective judgements favour our solution with respect to widely adopted atlas-based software.     

3D image segmentation of deformable objects with joint shape-intensity prior models using level sets

We propose a novel method for 3D image segmentation, where a Bayesian formulation, based on joint prior knowledge of the object shape and the image gray levels, along with information derived from the input image, is employed. Our method is motivated by the observation that the shape of an object and the gray level variation in an image have consistent relations that provide configurations and context that aid in segmentation. We define a maximum a posteriori (MAP) estimation model using the joint prior information of the object shape and the image gray levels to realize image segmentation. We introduce a representation for the joint density function of the object and the image gray level values, and define a joint probability distribution over the variations of the object shape and the gray levels contained in a set of training images. By estimating the MAP shape of the object, we formulate the shape-intensity model in terms of level set functions as opposed to landmark points of the object shape. In addition, we evaluate the performance of the level set representation of the object shape by comparing it with the point distribution model (PDM). We found the algorithm to be robust to noise and able to handle multidimensional data, while able to avoid the need for explicit point correspondences during the training phase. Results and validation from various experiments on 2D and 3D medical images are shown.     

Deformable organisms for automatic medical image analysis

We introduce a new approach to medical image analysis that combines deformable model methodologies with concepts from the field of artificial life. In particular, we propose "deformable organisms", autonomous agents whose task is the automatic segmentation, labeling, and quantitative analysis of anatomical structures in medical images. Analogous to natural organisms capable of voluntary movement, our artificial organisms possess deformable bodies with distributed sensors, as well as (rudimentary) brains with motor, perception, behavior, and cognition centers. Deformable organisms are perceptually aware of the image analysis process. Their behaviors, which manifest themselves in voluntary movement and alteration of body shape, are based upon sensed image features, pre-stored anatomical knowledge, and a deliberate cognitive plan. We demonstrate several prototype deformable organisms based on a multiscale axisymmetric body morphology, including a "corpus callosum worm" that can overcome noise, incomplete edges, considerable anatomical variation, and interference from collateral structures to segment and label the corpus callosum in 2D mid-sagittal MR brain images.