Cross contrast multi-channel image registration using image synthesis for MR brain images

Multi-modal deformable registration is important for many medical image analysis tasks such as atlas alignment, image fusion, and distortion correction. Whereas a conventional method would register images with different modalities using modality independent features or information theoretic metrics such as mutual information, this paper presents a new framework that addresses the problem using a two-channel registration algorithm capable of using mono-modal similarity measures such as sum of squared differences or cross-correlation. To make it possible to use these same-modality measures, image synthesis is used to create proxy images for the opposite modality as well as intensity-normalized images from each of the two available images. The new deformable registration framework was evaluated by performing intra-subject deformation recovery, intra-subject boundary alignment, and inter-subject label transfer experiments using multi-contrast magnetic resonance brain imaging data. Three different multi-channel registration algorithms were evaluated, revealing that the framework is robust to the multi-channel deformable registration algorithm that is used. With a single exception, all results demonstrated improvements when compared against single channel registrations using the same algorithm with mutual information.     

Multi-modal registration for correlative microscopy using image analogies

Correlative microscopy is a methodology combining the functionality of light microscopy with the high resolution of electron microscopy and other microscopy technologies for the same biological specimen. In this paper, we propose an image registration method for correlative microscopy, which is challenging due to the distinct appearance of biological structures when imaged with different modalities. Our method is based on image analogies and allows to transform images of a given modality into the appearance-space of another modality. Hence, the registration between two different types of microscopy images can be transformed to a mono-modality image registration. We use a sparse representation model to obtain image analogies. The method makes use of corresponding image training patches of two different imaging modalities to learn a dictionary capturing appearance relations. We test our approach on backscattered electron (BSE) scanning electron microscopy (SEM)/confocal and transmission electron microscopy (TEM)/confocal images. We perform rigid, affine, and deformable registration via B-splines and show improvements over direct registration using both mutual information and sum of squared differences similarity measures to account for differences in image appearance.     

Deep-learning based multi-modal retinal image registration for the longitudinal analysis of patients with age-related macular degeneration

This work reports a deep-learning based registration algorithm that aligns multi-modal retinal images collected from longitudinal clinical studies to achieve accuracy and robustness required for analysis of structural changes in large-scale clinical data. Deep-learning networks that mirror the architecture of conventional feature-point-based registration were evaluated with different networks that solved for registration affine parameters, image patch displacements, and patch displacements within the region of overlap. The ground truth images for deep learning-based approaches were derived from successful conventional feature-based registration. Cross-sectional and longitudinal affine registrations were performed across color fundus photography (CFP), fundus autofluorescence (FAF), and infrared reflectance (IR) image modalities. For mono-modality longitudinal registration, the conventional feature-based registration method achieved mean errors in the range of 39-53 µm (depending on the modality) whereas the deep learning method with region overlap prediction exhibited mean errors in the range 54-59 µm. For cross-sectional multi-modality registration, the conventional method exhibited gross failures with large errors in more than 50% of the cases while the proposed deep-learning method achieved robust performance with no gross failures and mean errors in the range 66-69 µm. Thus, the deep learning-based method achieved superior overall performance across all modalities. The accuracy and robustness reported in this work provide important advances that will facilitate clinical research and enable a detailed study of the progression of retinal diseases such as age-related macular degeneration.     

A unified framework for cross-modality multi-atlas segmentation of brain MRI

Multi-atlas label fusion is a powerful image segmentation strategy that is becoming increasingly popular in medical imaging. A standard label fusion algorithm relies on independently computed pairwise registrations between individual atlases and the (target) image to be segmented. These registrations are then used to propagate the atlas labels to the target space and fuse them into a single final segmentation. Such label fusion schemes commonly rely on the similarity between intensity values of the atlases and target scan, which is often problematic in medical imaging – in particular, when the atlases and target images are obtained via different sensor types or imaging protocols.In this paper, we present a generative probabilistic model that yields an algorithm for solving the atlas-to-target registrations and label fusion steps simultaneously. The proposed model does not directly rely on the similarity of image intensities. Instead, it exploits the consistency of voxel intensities within the target scan to drive the registration and label fusion, hence the atlases and target image can be of different modalities. Furthermore, the framework models the joint warp of all the atlases, introducing interdependence between the registrations.We use variational expectation maximization and the Demons registration framework in order to efficiently identify the most probable segmentation and registrations. We use two sets of experiments to illustrate the approach, where proton density (PD) MRI atlases are used to segment T1-weighted brain scans and vice versa. Our results clearly demonstrate the accuracy gain due to exploiting within-target intensity consistency and integrating registration into label fusion.     

MR to ultrasound registration for image-guided prostate interventions

A deformable registration method is described that enables automatic alignment of magnetic resonance (MR) and 3D transrectal ultrasound (TRUS) images of the prostate gland. The method employs a novel "model-to-image" registration approach in which a deformable model of the gland surface, derived from an MR image, is registered automatically to a TRUS volume by maximising the likelihood of a particular model shape given a voxel-intensity-based feature that represents an estimate of surface normal vectors at the boundary of the gland. The deformation of the surface model is constrained by a patient-specific statistical model of gland deformation, which is trained using data provided by biomechanical simulations. Each simulation predicts the motion of a volumetric finite element mesh due to the random placement of a TRUS probe in the rectum. The use of biomechanical modelling in this way also allows a dense displacement field to be calculated within the prostate, which is then used to non-rigidly warp the MR image to match the TRUS image. Using data acquired from eight patients, and anatomical landmarks to quantify the registration accuracy, the median final RMS target registration error after performing 100 MR-TRUS registrations for each patient was 2.40 mm.     

TransMorph: Transformer for unsupervised medical image registration

In the last decade, convolutional neural networks (ConvNets) have been a major focus of research in medical image analysis. However, the performances of ConvNets may be limited by a lack of explicit consideration of the long-range spatial relationships in an image. Recently, Vision Transformer architectures have been proposed to address the shortcomings of ConvNets and have produced state-of-the-art performances in many medical imaging applications. Transformers may be a strong candidate for image registration because their substantially larger receptive field enables a more precise comprehension of the spatial correspondence between moving and fixed images. Here, we present TransMorph, a hybrid Transformer-ConvNet model for volumetric medical image registration. This paper also presents diffeomorphic and Bayesian variants of TransMorph: the diffeomorphic variants ensure the topology-preserving deformations, and the Bayesian variant produces a well-calibrated registration uncertainty estimate. We extensively validated the proposed models using 3D medical images from three applications: inter-patient and atlas-to-patient brain MRI registration and phantom-to-CT registration. The proposed models are evaluated in comparison to a variety of existing registration methods and Transformer architectures. Qualitative and quantitative results demonstrate that the proposed Transformer-based model leads to a substantial performance improvement over the baseline methods, confirming the effectiveness of Transformers for medical image registration.     

Incorporating Prior Knowledge into Image Registration

The first step in the spatial normalization of brain images is usually to determine the affine transformation that best maps the image to a template image in a standard space. We have developed a rapid and automatic method for performing this registration, which uses a Bayesian scheme to incorporate prior knowledge of the variability in the shape and size of heads. We compared affine registrations with and without incorporating the prior knowledge. We found that the affine transformations derived using the Bayesian scheme are much more robust and that the rate of convergence is greater.     

Analysis of macular OCT images using deformable registration

Optical coherence tomography (OCT) of the macula has become increasingly important in the investigation of retinal pathology. However, deformable image registration, which is used for aligning subjects for pairwise comparisons, population averaging, and atlas label transfer, has not been well–developed and demonstrated on OCT images. In this paper, we present a deformable image registration approach designed specifically for macular OCT images. The approach begins with an initial translation to align the fovea of each subject, followed by a linear rescaling to align the top and bottom retinal boundaries. Finally, the layers within the retina are aligned by a deformable registration using one-dimensional radial basis functions. The algorithm was validated using manual delineations of retinal layers in OCT images from a cohort consisting of healthy controls and patients diagnosed with multiple sclerosis (MS). We show that the algorithm overcomes the shortcomings of existing generic registration methods, which cannot be readily applied to OCT images. A successful deformable image registration algorithm for macular OCT opens up a variety of population based analysis techniques that are regularly used in other imaging modalities, such as spatial normalization, statistical atlas creation, and voxel based morphometry. Examples of these applications are provided to demonstrate the potential benefits such techniques can have on our understanding of retinal disease. In particular, included is a pilot study of localized volumetric changes between healthy controls and MS patients using the proposed registration algorithm.OCIS codes: (100.0100) Image processing, (170.4470) Ophthalmology, (170.4500) Optical coherence tomography     

3D–2D registration in endovascular image-guided surgery: evaluation of state-of-the-art methods on cerebral angiograms

Purpose

Image guidance for minimally invasive surgery is based on spatial co-registration and fusion of 3D pre-interventional images and treatment plans with the 2D live intra-interventional images. The spatial co-registration or 3D–2D registration is the key enabling technology; however, the performance of state-of-the-art automated methods is rather unclear as they have not been assessed under the same test conditions. Herein we perform a quantitative and comparative evaluation of ten state-of-the-art methods for 3D–2D registration on a public dataset of clinical angiograms.

Methods

Image database consisted of 3D and 2D angiograms of 25 patients undergoing treatment for cerebral aneurysms or arteriovenous malformations. On each of the datasets, highly accurate “gold-standard” registrations of 3D and 2D images were established based on patient-attached fiducial markers. The database was used to rigorously evaluate ten state-of-the-art 3D–2D registration methods, namely two intensity-, two gradient-, three feature-based and three hybrid methods, both for registration of 3D pre-interventional image to monoplane or biplane 2D images.

Results

Intensity-based methods were most accurate in all tests (0.3 mm). One of the hybrid methods was most robust with 98.75% of successful registrations (SR) and capture range of 18 mm for registrations of 3D to biplane 2D angiograms. In general, registration accuracy was similar whether registration of 3D image was performed onto mono- or biplanar 2D images; however, the SR was substantially lower in case of 3D to monoplane 2D registration. Two feature-based and two hybrid methods had clinically feasible execution times in the order of a second.

Conclusions

Performance of methods seems to fall below expectations in terms of robustness in case of registration of 3D to monoplane 2D images, while translation into clinical image guidance systems seems readily feasible for methods that perform registration of the 3D pre-interventional image onto biplanar intra-interventional 2D images.

     

Piecewise-diffeomorphic image registration: Application to the motion estimation between 3D CT lung images with sliding conditions

In this paper, we propose a new strategy for modelling sliding conditions when registering 3D images in a piecewise-diffeomorphic framework. More specifically, our main contribution is the development of a mathematical formalism to perform Large Deformation Diffeomorphic Metric Mapping registration with sliding conditions. We also show how to adapt this formalism to the LogDemons diffeomorphic registration framework. We finally show how to apply this strategy to estimate the respiratory motion between 3D CT pulmonary images. Quantitative tests are performed on 2D and 3D synthetic images, as well as on real 3D lung images from the MICCAI EMPIRE10 challenge. Results show that our strategy estimates accurate mappings of entire 3D thoracic image volumes that exhibit a sliding motion, as opposed to conventional registration methods which are not capable of capturing discontinuous deformations at the thoracic cage boundary. They also show that although the deformations are not smooth across the location of sliding conditions, they are almost always invertible in the whole image domain. This would be helpful for radiotherapy planning and delivery.     

Semiautomatic registration of volumetric ultrasound scans.

We demonstrate the ability to register easily and accurately volumetric ultrasound scans without significant data preprocessing or user intervention. Two volumetric ultrasound breast scan data sets were acquired from two different patients with breast cancer. Volumetric scan data were acquired by manually sweeping a linear array transducer mounted on a linear slider with a position encoder. The volumetric data set pairs consisted of color flow and/or power mode Doppler data sets acquired serially on the same patients. A previously described semiautomatic registration method based on maximizing mutual information was used to determine the transform between data sets. The results suggest that, even for the deformable breast, three-dimensional full affine transforms can be sufficient to obtain clinically useful registrations; warping may be necessary for increased registration accuracy. In conclusion, mutual information-based automatic registration as implemented on modern workstations is capable of yielding clinically useful registrations in times <35 min.     

T-snakes: topology adaptive snakes

We present a new class of deformable contours (snakes) and apply them to the segmentation of medical images. Our snakes are defined in terms of an affine cell image decomposition (ACID). The 'snakes in ACID' framework significantly extends conventional snakes, enabling topological flexibility among other features. The resulting topology adaptive snakes, or 'T-snakes', can be used to segment some of the most complex-shaped biological structures from medical images in an efficient and highly automated manner.     

Hierarchical max-flow segmentation framework for multi-atlas segmentation with Kohonen self-organizing map based Gaussian mixture modeling

The incorporation of intensity, spatial, and topological information into large-scale multi-region segmentation has been a topic of ongoing research in medical image analysis. Multi-region segmentation problems, such as segmentation of brain structures, pose unique challenges in image segmentation in which regions may not have a defined intensity, spatial, or topological distinction, but rely on a combination of the three. We propose a novel framework within the Advanced segmentation tools (ASETS)², which combines large-scale Gaussian mixture models trained via Kohonen self-organizing maps, with deformable registration, and a convex max-flow optimization algorithm incorporating region topology as a hierarchy or tree. Our framework is validated on two publicly available neuroimaging datasets, the OASIS and MRBrainS13 databases, against the more conventional Potts model, achieving more accurate segmentations. Each component is accelerated using general-purpose programming on graphics processing Units to ensure computational feasibility.     

Statistical representation of high-dimensional deformation fields with application to statistically constrained 3D warping

This paper proposes a 3D statistical model aiming at effectively capturing statistics of high-dimensional deformation fields and then uses this prior knowledge to constrain 3D image warping. The conventional statistical shape model methods, such as the active shape model (ASM), have been very successful in modeling shape variability. However, their accuracy and effectiveness typically drop dramatically in high-dimensionality problems involving relatively small training datasets, which is customary in 3D and 4D medical imaging applications. The proposed statistical model of deformation (SMD) uses wavelet-based decompositions coupled with PCA in each wavelet band, in order to more accurately estimate the pdf of high-dimensional deformation fields, when a relatively small number of training samples are available. SMD is further used as statistical prior to regularize the deformation field in an SMD-constrained deformable registration framework. As a result, more robust registration results are obtained relative to using generic smoothness constraints on deformation fields, such as Laplacian-based regularization. In experiments, we first illustrate the performance of SMD in representing the variability of deformation fields and then evaluate the performance of the SMD-constrained registration, via comparing a hierarchical volumetric image registration algorithm, HAMMER, with its SMD-constrained version, referred to as SMD+HAMMER. This SMD-constrained deformable registration framework can potentially incorporate various registration algorithms to improve robustness and stability via statistical shape constraints.     

Atlas-ISTN: Joint segmentation,registration and atlas construction with image-and-spatial transformer networks

Deep learning models for semantic segmentation are able to learn powerful representations for pixel-wise predictions, but are sensitive to noise at test time and may lead to implausible topologies. Image registration models on the other hand are able to warp known topologies to target images as a means of segmentation, but typically require large amounts of training data, and have not widely been benchmarked against pixel-wise segmentation models. We propose the Atlas Image-and-Spatial Transformer Network (Atlas-ISTN), a framework that jointly learns segmentation and registration on 2D and 3D image data, and constructs a population-derived atlas in the process. Atlas-ISTN learns to segment multiple structures of interest and to register the constructed atlas labelmap to an intermediate pixel-wise segmentation. Additionally, Atlas-ISTN allows for test time refinement of the model’s parameters to optimize the alignment of the atlas labelmap to an intermediate pixel-wise segmentation. This process both mitigates for noise in the target image that can result in spurious pixel-wise predictions, as well as improves upon the one-pass prediction of the model. Benefits of the Atlas-ISTN framework are demonstrated qualitatively and quantitatively on 2D synthetic data and 3D cardiac computed tomography and brain magnetic resonance image data, out-performing both segmentation and registration baseline models. Atlas-ISTN also provides inter-subject correspondence of the structures of interest.     

Deformable registration of diffusion tensor MR images with explicit orientation optimization

In this paper, we present a novel deformable registration algorithm for diffusion tensor MR images that enables explicit optimization of tensor reorientation. The optimization seeks a piecewise affine transformation that divides the image domain into uniform regions and transform each region affinely. The objective function captures both the image similarity and the smoothness of the transformation across region boundaries. The image similarity enables explicit orientation optimization by incorporating tensor reorientation, which is necessary for warping diffusion tensor images. The objective function is formulated in a way that allows explicit implementation of analytic derivatives to drive fast and accurate optimization using the conjugate gradient method. By explicitly optimizing tensor reorientation, the algorithm is designed to take advantage of similarity measures comparing tensors as a whole. The optimal transformation is hierarchically refined in a subdivision framework. A comparison with affine registration for inter-subject normalization of 8 subjects shows that the proposed algorithm improves the alignment of several major white matter structures examined: the anterior thalamic radiations, the inferior fronto-occipital fasciculi, the corticospinal/corticobulbar tracts and the genu and the splenium of the corpus callosum. The alignment of white matter structures is assessed using a novel scheme of computing distances between the corresponding fiber bundles derived from tractography.     

Image registration: Maximum likelihood,minimum entropy and deep learning

In this work, we propose a theoretical framework based on maximum profile likelihood for pairwise and groupwise registration. By an asymptotic analysis, we demonstrate that maximum profile likelihood registration minimizes an upper bound on the joint entropy of the distribution that generates the joint image data. Further, we derive the congealing method for groupwise registration by optimizing the profile likelihood in closed form, and using coordinate ascent, or iterative model refinement. We also describe a method for feature based registration in the same framework and demonstrate it on groupwise tractographic registration. In the second part of the article, we propose an approach to deep metric registration that implements maximum likelihood registration using deep discriminative classifiers. We show further that this approach can be used for maximum profile likelihood registration to discharge the need for well-registered training data, using iterative model refinement. We demonstrate that the method succeeds on a challenging registration problem where the standard mutual information approach does not perform well.     

A Retrospective Analysis of Prostate Cone Beam Computed Tomography (CBCT) Image Registration: A Tale of Two Techniques

PurposeTo determine the feasibility of the gray-value registration technique as a correction method for prostate image-guided radiotherapy.BackgroundThe current practice at this institution involves a two-step process beginning with an automated bone registration, followed by a manual therapist manipulated registration.ProblemThis process is susceptible to subjective judgement and inter-observer variability, which introduces inconsistencies in image registrations and subsequent treatment delivery. Variation in the time required to complete image registrations may increase the chance for prostate intra-fractional movement, which may increase treatment inaccuracies.MethodsA retrospective image review was completed for 20 patients. Translational shifts and duration data for therapist registrations were retrieved for 15 scans for each patient, totaling 300 scans. Gray-value registrations were completed on the same 300 scans to recalculate shift values and the time to complete this technique was recorded. The analysis of variance test was used to analyze the extent of variance between the two registration techniques.ResultsNo difference between the two registration techniques in the lateral (right-left) direction (P = .364) and superoinferior directions (P = .455) were found. Gray-value registrations took significantly less time to complete and were more consistent than the two-step therapist registrations (P < .001). A statistically significant difference in z shifts exists (P = .003) between the two techniques.ConclusionGray-value registration can minimize and standardize the image registration durations and produce x and y shifts similar to those produced by therapists. Lacking the ability to preferentially match to the prostate-rectum interface, therapists still need to perform a simple inspection in the z (anteroposterior) plane to ensure an adequate registration to the interface.     

Image registration based on virtual frame sequence analysis

Purpose This paper is to propose a new framework for medical image registration with large nonrigid deformations, which still remains one of the biggest challenges for image fusion and further analysis in many medical applications. Method Registration problem is formulated as to recover a deformation process with the known initial state and final state. To deal with large nonlinear deformations, virtual frames are proposed to be inserted to model the deformation process. A time parameter is introduced and the deformation between consecutive frames is described with a linear affine transformation. Results Experiments are conducted with simple geometric deformation as well as complex deformations presented in MRI and ultrasound images. All the deformations are characterized with nonlinearity. The positive results demonstrated the effectiveness of this algorithm. Conclusion The framework proposed in this paper is feasible to register medical images with large nonlinear deformations and is especially useful for sequential images.