Spatio-temporal free-form registration of cardiac MR image sequences

In this paper we present two registration algorithms for the spatio-temporal alignment of cardiac MR image sequences. Both algorithms have the ability to correct spatial misalignment between the images sequences caused by global and local shape differences. In addition, they have the ability to correct temporal misalignment caused by differences in the length of the cardiac cycles and by differences in the dynamic properties of the hearts. The algorithms use a 4D deformable transformation model which is separated into spatial and temporal components. The first registration algorithm optimizes the spatial and temporal transformation models simultaneously, while the second registration algorithm optimizes the temporal transformation component before optimizing the spatial component. For the evaluation of the spatio-temporal registration methods we have acquired 15 MR image sequences from healthy volunteers. The registration methods were quantitatively evaluated by measuring the overlap and surface distance of anatomical regions and qualitatively by visual inspection. The results demonstrate that a significant improvement in the alignment of the image sequences is achieved by the use of the deformable spatio-temporal transformation model. We demonstrate the use of the method for the construction of a probabilistic MR cardiac atlas representing the anatomy and function of a healthy heart.     

Non-rigid multi-frame registration of cell nuclei in live cell fluorescence microscopy image data

The analysis of the motion of subcellular particles in live cell microscopy images is essential for understanding biological processes within cells. For accurate quantification of the particle motion, compensation of the motion and deformation of the cell nucleus is required. We introduce a non-rigid multi-frame registration approach for live cell fluorescence microscopy image data. Compared to existing approaches using pairwise registration, our approach exploits information from multiple consecutive images simultaneously to improve the registration accuracy. We present three intensity-based variants of the multi-frame registration approach and we investigate two different temporal weighting schemes. The approach has been successfully applied to synthetic and live cell microscopy image sequences, and an experimental comparison with non-rigid pairwise registration has been carried out.     

Reversible jump MCMC methods for fully automatic motion analysis in tagged MRI

Tagged magnetic resonance imaging (tMRI) is a well-known noninvasive method for studying regional heart dynamics. It offers great potential for quantitative analysis of a variety of kine(ma)tic parameters, but its clinical use has so far been limited, in part due to the lack of robustness and accuracy of existing tag tracking algorithms in dealing with low (and intrinsically time-varying) image quality. In this paper, we evaluate the performance of four frequently used concepts found in the literature (optical flow, harmonic phase (HARP) magnetic resonance imaging, active contour fitting, and non-rigid image registration) for cardiac motion analysis in 2D tMRI image sequences, using both synthetic image data (with ground truth) and real data from preclinical (small animal) and clinical (human) studies. In addition we propose a new probabilistic method for tag tracking that serves as a complementary step to existing methods. The new method is based on a Bayesian estimation framework, implemented by means of reversible jump Markov chain Monte Carlo (MCMC) methods, and combines information about the heart dynamics, the imaging process, and tag appearance. The experimental results demonstrate that the new method improves the performance of even the best of the four previous methods. Yielding higher consistency, accuracy, and intrinsic tag reliability assessment, the proposed method allows for improved analysis of cardiac motion.     

DIRBoost–An algorithm for boosting deformable image registration: Application to lung CT intra-subject registration

We introduce a boosting algorithm to improve on existing methods for deformable image registration (DIR). The proposed DIRBoost algorithm is inspired by the theory on hypothesis boosting, well known in the field of machine learning. DIRBoost utilizes a method for automatic registration error detection to obtain estimates of local registration quality. All areas detected as erroneously registered are subjected to boosting, i.e. undergo iterative registrations by employing boosting masks on both the fixed and moving image. We validated the DIRBoost algorithm on three different DIR methods (ANTS gSyn, NiftyReg, and DROP) on three independent reference datasets of pulmonary image scan pairs. DIRBoost reduced registration errors significantly and consistently on all reference datasets for each DIR algorithm, yielding an improvement of the registration accuracy by 5–34% depending on the dataset and the registration algorithm employed.     

Registration of longitudinal brain image sequences with implicit template and spatial-temporal heuristics

Accurate measurement of longitudinal changes of brain structures and functions is very important but challenging in many clinical studies. Also, across-subject comparison of longitudinal changes is critical in identifying disease-related changes. In this paper, we propose a novel method to meet these two requirements by simultaneously registering sets of longitudinal image sequences of different subjects to the common space, without assuming any explicit template. Specifically, our goal is to 1) consistently measure the longitudinal changes from a longitudinal image sequence of each subject, and 2) jointly align all image sequences of different subjects to a hidden common space. To achieve these two goals, we first introduce a set of temporal fiber bundles to explore the spatial-temporal behavior of anatomical changes in each longitudinal image sequence. Then, a probabilistic model is built upon the temporal fibers to characterize both spatial smoothness and temporal continuity. Finally, the transformation fields that connect each time-point image of each subject to the common space are simultaneously estimated by the expectation maximization (EM) approach, via the maximum a posterior (MAP) estimation of the probabilistic models. Promising results have been obtained in quantitative measurement of longitudinal brain changes, i.e., hippocampus volume changes, showing better performance than those obtained by either the pairwise or the groupwise only registration methods.     

Spectral-spatial classification of hyperspectral imagery based on recurrent neural networks

Deep neural networks have recently been successfully explored to extract deep features for hyperspectral image classification. Recurrent neural networks (RNNs) are an important branch of the deep learning family, which are widely used for sequence analysis. Indeed, RNNs have been used to model the dependencies between the different spectral bands of hyperspectral image, inspired by the observation that hyperspectral pixels can be considered as spectral sequences. A disadvantage of such methods is that they don’t consider the effect of neighborhood pixels on the final class label. In this letter, a RNN model is proposed for the spectral-spatial classification of hyperspectral image. Specifically, the hyperspectral image cube surrounding a central pixel is considered as a hyperspectral pixels sequence, and a RNN is used to model the dependencies between the different neighborhood pixels. The proposed RNN is conducted on two widely used hyperspectral image datasets. The experimental results demonstrate that the proposed approach provides a better performance than that of conventional methods.     

Deformable image registration for temporal subtraction of chest radiographs

Purpose

Temporal subtraction images constructed from image registration can facilitate the visualization of pathologic changes. In this study, we propose a deformable image registration (DIR) framework for creating temporal subtraction images of chest radiographs.

Methods

We developed a DIR methodology using two different image similarity metrics, varying flow (VF) and compressible flow (CF). The proposed registration method consists of block matching, filtering, and interpolation. Specifically, corresponding point pairs between reference and target images are initially determined by minimizing a nonlinear least squares formulation using grid-searching optimization. A two-step filtering process, including least median of squares filtering and backward matching filtering, is then applied to the estimated point matches in order to remove erroneous matches. Finally, moving least squares is used to generate a full displacement field from the filtered point pairs.

Results

We applied the proposed DIR method to 10 pairs of clinical chest radiographs and compared it with the demons and B-spline algorithms using the five-point rating score method. The average quality scores were 2.7 and 3 for the demons and B-spline methods, but 3.5 and 4.1 for the VF and CF methods. In addition, subtraction images improved the visual perception of abnormalities in the lungs by using the proposed method.

Conclusion

The VF and CF models achieved a higher accuracy than the demons and the B-spline methods. Furthermore, the proposed methodology demonstrated the ability to create clinically acceptable temporal subtraction chest radiographs that enhance interval changes and can be used to detect abnormalities such as non-small cell lung cancer.     

A dynamic elastic model for segmentation and tracking of the heart in MR image sequences

Strong prior models are a prerequisite for reliable spatio-temporal cardiac image analysis. While several cardiac models have been presented in the past, many of them are either too complex for their parameters to be estimated on the sole basis of MR Images, or overly simplified. In this paper, we present a novel dynamic model, based on the equation of dynamics for elastic materials and on Fourier filtering. The explicit use of dynamics allows us to enforce periodicity and temporal smoothness constraints. We propose an algorithm to solve the continuous dynamical problem associated to numerically adapting the model to the image sequence. Using a simple 1D example, we show how temporal filtering can help removing noise while ensuring the periodicity and smoothness of solutions. The proposed dynamic model is quantitatively evaluated on a database of 15 patients which shows its performance and limitations. Also, the ability of the model to capture cardiac motion is demonstrated on synthetic cardiac sequences. Moreover, existence, uniqueness of the solution and numerical convergence of the algorithm can be demonstrated.     

Stochastic finite element framework for simultaneous estimation of cardiac kinematic functions and material parameters

A stochastic finite element framework is presented for the simultaneous estimation of the cardiac kinematic functions and material model parameters from periodic medical image sequences. While existing biomechanics studies of the myocardial material constitutive laws have assumed known tissue kinematic measurements, and image analysis efforts on cardiac kinematic functions have relied on fixed constraining models of mathematical or mechanical nature, we illustrate through synthetic data that a probabilistic joint estimation strategy is needed to achieve more robust and accurate analysis of the kinematic functions and material parameters at the same time. For a particular a priori constraining material model with uncertain subject-dependent parameters and a posteriori noisy imaging based observations, our strategy combines the stochastic differential equations of the myocardial dynamics with the finite element method, and the material parameters and the imaging data are treated as random variables with known prior statistics. After the conversion to state space representation, the extended Kalman filtering procedures are adopted to linearize the equations and to provide the joint estimates in an approximate optimal sense. The estimation bias and convergence issues are addressed, and we conclude experimentally that it is possible to adopt this biomechanical model based multiframe estimation approach to achieve converged estimates because of the periodic nature of the cardiac dynamics. The effort is validated using synthetic data sequence with known kinematics and material parameters. Further, under linear elastic material model, estimation results using canine magnetic resonance phase contrast image sequences are presented, which are in very good agreement with histological tissue staining results, the current gold standards.     

Hidden Markov multiple event sequence models: A paradigm for the spatio-temporal analysis of fMRI data

This paper presents a novel, completely unsupervised fMRI brain mapping method that addresses the three problems of hemodynamic response function (HRF) variability, hemodynamic event timing, and fMRI response non-linearity. Spatial and temporal information are directly taken into account into the core of the activation detection process. In practice, activation detection at voxel v is formulated in terms of temporal alignment between sequences of hemodynamic response onsets (HROs) detected in the fMRI signal at v and in the spatial neighborhood of v, and the input sequence of stimuli or stimulus onsets. Event-related and epoch paradigms are considered. The multiple event sequence alignment problem is solved within the probabilistic framework of hidden Markov multiple event sequence models (HMMESMs), a new class of hidden Markov models. Results obtained on real and synthetic data significantly outperform those obtained with the popular statistical parametric mapping (SPM2) method without requiring any prior definition of the expected activation patterns, the HMMESM mapping approach being completely unsupervised.     

Temporal sparse free-form deformations

FFD represent a widely used model for the non-rigid registration of medical images. The balance between robustness to noise and accuracy in modelling localised motion is typically controlled by the control point grid spacing and the amount of regularisation. More recently, TFFD have been proposed which extend the FFD approach in order to recover smooth motion from temporal image sequences. In this paper, we revisit the classic FFD approach and propose a sparse representation using the principles of compressed sensing. The sparse representation can model both global and local motion accurately and robustly. We view the registration as a deformation reconstruction problem. The deformation is reconstructed from a pair of images (or image sequences) with a sparsity constraint applied to the parametric space. Specifically, we introduce sparsity into the deformation via L₁ regularisation, and apply a bending energy regularisation between neighbouring control points within each level to encourage a grouped sparse solution. We further extend the sparsity constraint to the temporal domain and propose a TSFFD which can capture fine local details such as motion discontinuities in both space and time without sacrificing robustness. We demonstrate the capabilities of the proposed framework to accurately estimate deformations in dynamic 2D and 3D image sequences. Compared to the classic FFD and TFFD approach, a significant increase in registration accuracy can be observed in natural images as well as in cardiac images.     

Probabilistic inference of regularisation in non-rigid registration

A long-standing issue in non-rigid image registration is the choice of the level of regularisation. Regularisation is necessary to preserve the smoothness of the registration and penalise against unnecessary complexity. The vast majority of existing registration methods use a fixed level of regularisation, which is typically hand-tuned by a user to provide “nice" results. However, the optimal level of regularisation will depend on the data which is being processed; lower signal-to-noise ratios require higher regularisation to avoid registering image noise as well as features, and different pairs of images require registrations of varying complexity depending on their anatomical similarity. In this paper we present a probabilistic registration framework that infers the level of regularisation from the data. An additional benefit of this proposed probabilistic framework is that estimates of the registration uncertainty are obtained. This framework has been implemented using a free-form deformation transformation model, although it would be generically applicable to a range of transformation models. We demonstrate our registration framework on the application of inter-subject brain registration of healthy control subjects from the NIREP database. In our results we show that our framework appropriately adapts the level of regularisation in the presence of noise, and that inferring regularisation on an individual basis leads to a reduction in model over-fitting as measured by image folding while providing a similar level of overlap.     

Deep probabilistic tracking of particles in fluorescence microscopy images

Tracking of particles in temporal fluorescence microscopy image sequences is of fundamental importance to quantify dynamic processes of intracellular structures as well as virus structures. We introduce a probabilistic deep learning approach for fluorescent particle tracking, which is based on a recurrent neural network that mimics classical Bayesian filtering. Compared to previous deep learning methods for particle tracking, our approach takes into account uncertainty, both aleatoric and epistemic uncertainty. Thus, information about the reliability of the computed trajectories is determined. Manual tuning of tracking parameters is not necessary and prior knowledge about the noise statistics is not required. Short and long-term temporal dependencies of individual object dynamics are exploited for state prediction, and assigned detections are used to update the predicted states. For correspondence finding, we introduce a neural network which computes assignment probabilities jointly across multiple detections as well as determines the probabilities of missing detections. Training requires only simulated data and therefore tedious manual annotation of ground truth is not needed. We performed a quantitative performance evaluation based on synthetic and real 2D as well as 3D fluorescence microscopy images. We used image data of the Particle Tracking Challenge as well as real time-lapse fluorescence microscopy images displaying virus structures and chromatin structures. It turned out that our approach yields state-of-the-art results or improves the tracking results compared to previous methods.     

Robust endoscopic image mosaicking via fusion of multimodal estimation

We propose an endoscopic image mosaicking algorithm that is robust to light conditioning changes, specular reflections, and feature-less scenes. These conditions are especially common in minimally invasive surgery where the light source moves with the camera to dynamically illuminate close range scenes. This makes it difficult for a single image registration method to robustly track camera motion and then generate consistent mosaics of the expanded surgical scene across different and heterogeneous environments. Instead of relying on one specialised feature extractor or image registration method, we propose to fuse different image registration algorithms according to their uncertainties, formulating the problem as affine pose graph optimisation. This allows to combine landmarks, dense intensity registration, and learning-based approaches in a single framework. To demonstrate our application we consider deep learning-based optical flow, hand-crafted features, and intensity-based registration, however, the framework is general and could take as input other sources of motion estimation, including other sensor modalities. We validate the performance of our approach on three datasets with very different characteristics to highlighting its generalisability, demonstrating the advantages of our proposed fusion framework. While each individual registration algorithm eventually fails drastically on certain surgical scenes, the fusion approach flexibly determines which algorithms to use and in which proportion to more robustly obtain consistent mosaics.     

A multi-scale probabilistic network model for detection,synthesis and compression in mammographic image analysis

We develop a probabilistic network model over image spaces and demonstrate its broad utility in mammographic image analysis, particularly with respect to computer-aided diagnosis. The model employs a multi-scale pyramid decomposition to factor images across scale and a network of tree-structured hidden variables to capture long-range spatial dependencies. This factoring makes the computation of the density functions local and tractable. The result is a hierarchical mixture of conditional probabilities, similar to a hidden Markov model on a tree. The model parameters are found with maximum likelihood estimation using the expectation-maximization algorithm. The utility of the model is demonstrated for three applications: (1) detection of mammographic masses for computer-aided diagnosis; (2) qualitative assessment of model structure through mammographic synthesis; and (3) compression of mammographic regions of interest.     

Piecewise-diffeomorphic image registration: Application to the motion estimation between 3D CT lung images with sliding conditions

In this paper, we propose a new strategy for modelling sliding conditions when registering 3D images in a piecewise-diffeomorphic framework. More specifically, our main contribution is the development of a mathematical formalism to perform Large Deformation Diffeomorphic Metric Mapping registration with sliding conditions. We also show how to adapt this formalism to the LogDemons diffeomorphic registration framework. We finally show how to apply this strategy to estimate the respiratory motion between 3D CT pulmonary images. Quantitative tests are performed on 2D and 3D synthetic images, as well as on real 3D lung images from the MICCAI EMPIRE10 challenge. Results show that our strategy estimates accurate mappings of entire 3D thoracic image volumes that exhibit a sliding motion, as opposed to conventional registration methods which are not capable of capturing discontinuous deformations at the thoracic cage boundary. They also show that although the deformations are not smooth across the location of sliding conditions, they are almost always invertible in the whole image domain. This would be helpful for radiotherapy planning and delivery.     

Multiview diffeomorphic registration: Application to motion and strain estimation from 3D echocardiography

This paper presents a new registration framework for quantifying myocardial motion and strain from the combination of multiple 3D ultrasound (US) sequences. The originality of our approach lies in the estimation of the transformation directly from the input multiple views rather than from a single view or a reconstructed compounded sequence. This allows us to exploit all spatiotemporal information available in the input views avoiding occlusions and image fusion errors that could lead to some inconsistencies in the motion quantification result.We propose a multiview diffeomorphic registration strategy that enforces smoothness and consistency in the spatiotemporal domain by modeling the 4D velocity field continuously in space and time. This 4D continuous representation considers 3D US sequences as a whole, therefore allowing to robustly cope with variations in heart rate resulting in different number of images acquired per cardiac cycle for different views. This contributes to the robustness gained by solving for a single transformation from all input sequences. The similarity metric takes into account the physics of US images and uses a weighting scheme to balance the contribution of the different views. It includes a comparison both between consecutive images and between a reference and each of the following images. The strain tensor is computed locally using the spatial derivatives of the reconstructed displacement fields.Registration and strain accuracy were evaluated on synthetic 3D US sequences with known ground truth. Experiments were also conducted on multiview 3D datasets of 8 volunteers and 1 patient treated by cardiac resynchronization therapy. Strain curves obtained from our multiview approach were compared to the single-view case, as well as with other multiview approaches. For healthy cases, the inclusion of several views improved the consistency of the strain curves and reduced the number of segments where a non-physiological strain pattern was observed. For the patient, the improvement (pacing ON vs. OFF) in synchrony of regional strain correlated with clinician blind assessment and could be seen more clearly when using the multiview approach.     

2-D registration and 3-D shape inference of the retinal fundus from fluorescein images

This study presents methods to 2-D registration of retinal image sequences and 3-D shape inference from fluorescein images. The Y-feature is a robust geometric entity that is largely invariant across modalities as well as across the temporal grey level variations induced by the propagation of the dye in the vessels. We first present a Y-feature extraction method that finds a set of Y-feature candidates using local image gradient information. A gradient-based approach is then used to align an articulated model of the Y-feature to the candidates more accurately while optimizing a cost function. Using mutual information, fitted Y-features are subsequently matched across images, including colors and fluorescein angiographic frames, for registration. To reconstruct the retinal fundus in 3-D, the extracted Y-features are used to estimate the epipolar geometry with a plane-and-parallax approach. The proposed solution provides a robust estimation of the fundamental matrix suitable for plane-like surfaces, such as the retinal fundus. The mutual information criterion is used to accurately estimate the dense disparity map. Our experimental results validate the proposed method on a set of difficult fluorescein image pairs.