Cross-modal attention for multi-modal image registration

In the past few years, convolutional neural networks (CNNs) have been proven powerful in extracting image features crucial for medical image registration. However, challenging applications and recent advances in computer vision suggest that CNNs are limited in their ability to understand the spatial correspondence between features, which is at the core of image registration. The issue is further exaggerated when it comes to multi-modal image registration, where the appearances of input images can differ significantly. This paper presents a novel cross-modal attention mechanism for correlating features extracted from the multi-modal input images and mapping such correlation to image registration transformation. To efficiently train the developed network, a contrastive learning-based pre-training method is also proposed to aid the network in extracting high-level features across the input modalities for the following cross-modal attention learning. We validated the proposed method on transrectal ultrasound (TRUS) to magnetic resonance (MR) registration, a clinically important procedure that benefits prostate cancer biopsy. Our experimental results demonstrate that for MR-TRUS registration, a deep neural network embedded with the cross-modal attention block outperforms other advanced CNN-based networks with ten times its size. We also incorporated visualization techniques to improve the interpretability of our network, which helps bring insights into the deep learning based image registration methods. The source code of our work is available at https://github.com/DIAL-RPI/Attention-Reg.     

Dual-core steered non-rigid registration for multi-modal images via bi-directional image synthesis

In prostate cancer radiotherapy, computed tomography (CT) is widely used for dose planning purposes. However, because CT has low soft tissue contrast, it makes manual contouring difficult for major pelvic organs. In contrast, magnetic resonance imaging (MRI) provides high soft tissue contrast, which makes it ideal for accurate manual contouring. Therefore, the contouring accuracy on CT can be significantly improved if the contours in MRI can be mapped to CT domain by registering MRI with CT of the same subject, which would eventually lead to high treatment efficacy. In this paper, we propose a bi-directional image synthesis based approach for MRI-to-CT pelvic image registration. First, we use patch-wise random forest with auto-context model to learn the appearance mapping from CT to MRI domain, and then vice versa. Consequently, we can synthesize a pseudo-MRI whose anatomical structures are exactly same with CT but with MRI-like appearance, and a pseudo-CT as well. Then, our MRI-to-CT registration can be steered in a dual manner, by simultaneously estimating two deformation pathways: 1) one from the pseudo-CT to the actual CT and 2) another from actual MRI to the pseudo-MRI. Next, a dual-core deformation fusion framework is developed to iteratively and effectively combine these two registration pathways by using complementary information from both modalities. Experiments on a dataset with real pelvic CT and MRI have shown improved registration performance of the proposed method by comparing it to the conventional registration methods, thus indicating its high potential of translation to the routine radiation therapy.     

Weakly-supervised learning of multi-modal features for regularised iterative descent in 3D image registration

Methods for deep learning based medical image registration have only recently approached the quality of classical model-based image alignment. The dual challenge of both a very large trainable parameter space and often insufficient availability of expert supervised correspondence annotations has led to slower progress compared to other domains such as image segmentation. Yet, image registration could also more directly benefit from an iterative solution than segmentation. We therefore believe that significant improvements, in particular for multi-modal registration, can be achieved by disentangling appearance-based feature learning and deformation estimation. In this work, we examine an end-to-end trainable, weakly-supervised deep learning-based feature extraction approach that is able to map the complex appearance to a common space. Our results on thoracoabdominal CT and MRI image registration show that the proposed method compares favourably well to state-of-the-art hand-crafted multi-modal features, Mutual Information-based approaches and fully-integrated CNN-based methods - and handles even the limitation of small and only weakly-labeled training data sets.     

Mutual information based multi-modal remote sensing image registration using adaptive feature weight

Multi-module images registration is a challenging task in image processing, and more especially in the field of remote sensing. In this letter, we strive to present a novel mutual information scheme for image registration in remote sensing scenario based on feature map technique. We firstly take saliency detection advantages to extract geographic pattern, and then utilize the efficient Laplacian of Gaussian(LOG) and Guided Filter methods to construct a new feature map based on different characteristic of multi-channel images. To avoid practical traps of sub-optimization, we propose an novel mutual information(MI) algorithm based on an adapted weight strategy. The proposed model divides an image into patches and assigns weighted values according to patch similarities in order to solve the optimization problem, improve accuracy and enhance performance. Note that, our proposed method incorporates the LOG and Guided Filter methods into image registration for the first time to construct a new feature map based on differences and similarities strategy. Experiments are conducted over island and coastline scenes, and reveal that our hybrid model has a significant performance and outperforms the state-of-the-art methods in remote sensing image registration.     

Cross contrast multi-channel image registration using image synthesis for MR brain images

Multi-modal deformable registration is important for many medical image analysis tasks such as atlas alignment, image fusion, and distortion correction. Whereas a conventional method would register images with different modalities using modality independent features or information theoretic metrics such as mutual information, this paper presents a new framework that addresses the problem using a two-channel registration algorithm capable of using mono-modal similarity measures such as sum of squared differences or cross-correlation. To make it possible to use these same-modality measures, image synthesis is used to create proxy images for the opposite modality as well as intensity-normalized images from each of the two available images. The new deformable registration framework was evaluated by performing intra-subject deformation recovery, intra-subject boundary alignment, and inter-subject label transfer experiments using multi-contrast magnetic resonance brain imaging data. Three different multi-channel registration algorithms were evaluated, revealing that the framework is robust to the multi-channel deformable registration algorithm that is used. With a single exception, all results demonstrated improvements when compared against single channel registrations using the same algorithm with mutual information.     

Deep-learning based multi-modal retinal image registration for the longitudinal analysis of patients with age-related macular degeneration

This work reports a deep-learning based registration algorithm that aligns multi-modal retinal images collected from longitudinal clinical studies to achieve accuracy and robustness required for analysis of structural changes in large-scale clinical data. Deep-learning networks that mirror the architecture of conventional feature-point-based registration were evaluated with different networks that solved for registration affine parameters, image patch displacements, and patch displacements within the region of overlap. The ground truth images for deep learning-based approaches were derived from successful conventional feature-based registration. Cross-sectional and longitudinal affine registrations were performed across color fundus photography (CFP), fundus autofluorescence (FAF), and infrared reflectance (IR) image modalities. For mono-modality longitudinal registration, the conventional feature-based registration method achieved mean errors in the range of 39-53 µm (depending on the modality) whereas the deep learning method with region overlap prediction exhibited mean errors in the range 54-59 µm. For cross-sectional multi-modality registration, the conventional method exhibited gross failures with large errors in more than 50% of the cases while the proposed deep-learning method achieved robust performance with no gross failures and mean errors in the range 66-69 µm. Thus, the deep learning-based method achieved superior overall performance across all modalities. The accuracy and robustness reported in this work provide important advances that will facilitate clinical research and enable a detailed study of the progression of retinal diseases such as age-related macular degeneration.     

混合角点检测算法用于脑磁共振图像配准

目的针对现有角点检测算法的不足,提出结合Harris、Susan的混合角点检测算法,并应用于脑MR图像配准中。方法首先通过Harris算子、Susan算子分别提取图像中Harris角点和Susan角点;然后对Harris角点和Susan角点取并集;通过引入两个加权因子ω1和ω2,分别对Harris角点响应值与Susan角点响应值进行加权计算,获得其角点强度,从而筛选出新的角点集合;通过归一化相关法和投票策略筛选出精确匹配的角点对;最后采用Powell算法进一步优化,获得图像最终配准参数值。结果混合角点检测算法应用于脑MR图像配准能获得较高的配准精度和较好的稳定性。结论相比于目前的角点检测算法,本文算法更适用于脑MR图像配准。     

Entropy and Laplacian images: structural representations for multi-modal registration

The standard approach to multi-modal registration is to apply sophisticated similarity metrics such as mutual information. The disadvantage of these metrics, in comparison to measuring the intensity difference with, e.g. L1 or L2 distance, is the increase in computational complexity and consequently the increase in runtime of the registration. An alternative approach, which has not yet gained much attention in the literature, is to find image representations, so called structural representations, that allow for the application of the L1 and L2 distance for multi-modal images. This has not only the advantage of a faster similarity calculation but enables also the application of more sophisticated optimization strategies. In this article, we theoretically analyze the requirements for structural representations. Further, we introduce two approaches to create such representations, which are based on the calculation of patch entropy and manifold learning, respectively. While the application of entropy has practical advantages in terms of computational complexity, the usage of manifold learning has theoretical advantages, by presenting an optimal approximation to one of the theoretical requirements. We perform experiments on multiple datasets for rigid, deformable, and groupwise registration with good results with respect to both, runtime and quality of alignment.     

Improved optimization for the robust and accurate linear registration and motion correction of brain images

Linear registration and motion correction are important components of structural and functional brain image analysis. Most modern methods optimize some intensity-based cost function to determine the best registration. To date, little attention has been focused on the optimization method itself, even though the success of most registration methods hinges on the quality of this optimization. This paper examines the optimization process in detail and demonstrates that the commonly used multiresolution local optimization methods can, and do, get trapped in local minima. To address this problem, two approaches are taken: (1) to apodize the cost function and (2) to employ a novel hybrid global-local optimization method. This new optimization method is specifically designed for registering whole brain images. It substantially reduces the likelihood of producing misregistrations due to being trapped by local minima. The increased robustness of the method, compared to other commonly used methods, is demonstrated by a consistency test. In addition, the accuracy of the registration is demonstrated by a series of experiments with motion correction. These motion correction experiments also investigate how the results are affected by different cost functions and interpolation methods.     

三维多模态医学图像自动配准方法研究

提出一种综合应用图像分割与互信息的医学图像自动配准方法.首先采用门限法和数学形态学方法进行预处理,再用k-means方法进行分割,之后采用基于互信息的Powell优化方法配准.将该方法用于磁共振图像(MRI)和正电子发射断层扫描(PET)临床医学图像配准,得到较满意的效果.     

A survey of medical image registration

The purpose of this paper is to present a survey of recent (published in 1993 or later) publications concerning medical image registration techniques. These publications will be classified according to a model based on nine salient criteria, the main dichotomy of which is extrinsic versus intrinsic methods. The statistics of the classification show definite trends in the evolving registration techniques, which will be discussed. At this moment, the bulk of interesting intrinsic methods is based on either segmented points or surfaces, or on techniques endeavouring to use the full information content of the images involved.     

A fast diffeomorphic image registration algorithm

This paper describes DARTEL, which is an algorithm for diffeomorphic image registration. It is implemented for both 2D and 3D image registration and has been formulated to include an option for estimating inverse consistent deformations. Nonlinear registration is considered as a local optimisation problem, which is solved using a Levenberg-Marquardt strategy. The necessary matrix solutions are obtained in reasonable time using a multigrid method. A constant Eulerian velocity framework is used, which allows a rapid scaling and squaring method to be used in the computations. DARTEL has been applied to intersubject registration of 471 whole brain images, and the resulting deformations were evaluated in terms of how well they encode the shape information necessary to separate male and female subjects and to predict the ages of the subjects.     

Maximum distance-gradient for robust image registration

To make up for the lack of concern on the spatial information in the conventional mutual information based image registration framework, this paper designs a novel spatial feature field, namely the maximum distance-gradient (MDG) vector field, for registration tasks. It encodes both the local edge information and globally defined spatial information related to the intensity difference, the distance, and the direction of a voxel to a MDG source point. A novel similarity measure is proposed as the combination of the multi-dimensional mutual information and an angle measure on the MDG vector field. This measure integrates both the magnitude and orientation information of the MDG vector field into the image registration process.Experimental results on clinical 3D CT and T1-weighted MR image volumes show that, as compared with the conventional mutual information based method and two of its adaptations incorporating spatial information, the proposed method can give longer capture ranges at different image resolutions. This leads to more robust registrations. Around 2000 randomized rigid registration experiments demonstrate that our method consistently gives much higher success rates than the aforementioned three related methods. Moreover, it is shown that the registration accuracy of our method is high.     

Semantic similarity metrics for image registration

Image registration aims to find geometric transformations that align images. Most algorithmic and deep learning-based methods solve the registration problem by minimizing a loss function, consisting of a similarity metric comparing the aligned images, and a regularization term ensuring smoothness of the transformation. Existing similarity metrics like Euclidean Distance or Normalized Cross-Correlation focus on aligning pixel intensity values or correlations, giving difficulties with low intensity contrast, noise, and ambiguous matching. We propose a semantic similarity metric for image registration, focusing on aligning image areas based on semantic correspondence instead. Our approach learns dataset-specific features that drive the optimization of a learning-based registration model. We train both an unsupervised approach extracting features with an auto-encoder, and a semi-supervised approach using supplemental segmentation data. We validate the semantic similarity metric using both deep-learning-based and algorithmic image registration methods. Compared to existing methods across four different image modalities and applications, the method achieves consistently high registration accuracy and smooth transformation fields.     

More accurate Talairach coordinates for neuroimaging using non-linear registration

While the Talairach atlas remains the most commonly used system for reporting coordinates in neuroimaging studies, the absence of an actual 3-D image of the original brain used in its construction has severely limited the ability of researchers to automatically map locations from 3-D anatomical MRI images to the atlas. Previous work in this area attempted to circumvent this problem by constructing approximate linear and piecewise-linear mappings between standard brain templates (e.g. the MNI template) and Talairach space. These methods are limited in that they can only account for differences in overall brain size and orientation but cannot correct for the actual shape differences between the MNI template and the Talairach brain. In this paper we describe our work to digitize the Talairach atlas and generate a non-linear mapping between the Talairach atlas and the MNI template that attempts to compensate for the actual differences in shape between the two, resulting in more accurate coordinate transformations. We present examples in this paper and note that the method is available freely online as a Java applet.     

A unified non-rigid feature registration method for brain mapping

This paper describes the design, implementation and results of a unified non-rigid feature registration method for the purposes of anatomical MRI brain registration. An important characteristic of the method is its ability to fuse different types of anatomical features into a single point-set representation. We demonstrate the application of the method using two different types of features: the outer cortical surface and major sulcal ribbons. Non-rigid registration of the combined feature point-sets is then performed using a new robust non-rigid point matching algorithm. The point matching algorithm implements an iterative joint clustering and matching (JCM) strategy which effectively reduces the computational complexity without sacrificing accuracy. We have conducted carefully designed synthetic experiments to gauge the effect of using different types of features either separately or together. A validation study examining the accuracy of non-rigid alignment of many brain structures is also presented. Finally, we present anecdotal results on the alignment of two subject MRI brain data.     

A multi-modal approach to computer-assisted deep brain stimulation trajectory planning

Purpose

Both frame-based and frameless approaches to deep brain stimulation (DBS) require planning of insertion trajectories that mitigate hemorrhagic risk and loss of neurological function. Currently, this is done by manual inspection of multiple potential electrode trajectories on MR-imaging data. We propose and validate a method for computer-assisted DBS trajectory planning.

Method

Our framework integrates multi-modal MRI analysis (T1w, SWI, TOF-MRA) to compute suitable DBS trajectories that optimize the avoidance of specific critical brain structures. A cylinder model is used to process each trajectory and to evaluate complex surgical constraints described via a combination of binary and fuzzy segmented datasets. The framework automatically aggregates the multiple constraints into a unique ranking of recommended low-risk trajectories. Candidate trajectories are represented as a few well-defined cortical entry patches of best-ranked trajectories and presented to the neurosurgeon for final trajectory selection.

Results

The proposed algorithm permits a search space containing over 8,000 possible trajectories to be processed in less than 20 s. A retrospective analysis on 14 DBS cases of patients with severe Parkinson’s disease reveals that our framework can improve the simultaneous optimization of many pre-formulated surgical constraints. Furthermore, all automatically computed trajectories were evaluated by two neurosurgeons, were judged suitable for surgery and, in many cases, were judged preferable or equivalent to the manually planned trajectories used during the operation.

Conclusions

This work provides neurosurgeons with an intuitive and flexible decision-support system that allows objective and patient-specific optimization of DBS lead trajectories, which should improve insertion safety and reduce surgical time.

     

Predict brain MR image registration via sparse learning of appearance and transformation

We propose a new approach to register the subject image with the template by leveraging a set of intermediate images that are pre-aligned to the template. We argue that, if points in the subject and the intermediate images share similar local appearances, they may have common correspondence in the template. In this way, we learn the sparse representation of a certain subject point to reveal several similar candidate points in the intermediate images. Each selected intermediate candidate can bridge the correspondence from the subject point to the template space, thus predicting the transformation associated with the subject point at the confidence level that relates to the learned sparse coefficient. Following this strategy, we first predict transformations at selected key points, and retain multiple predictions on each key point, instead of allowing only a single correspondence. Then, by utilizing all key points and their predictions with varying confidences, we adaptively reconstruct the dense transformation field that warps the subject to the template. We further embed the prediction–reconstruction protocol above into a multi-resolution hierarchy. In the final, we refine our estimated transformation field via existing registration method in effective manners. We apply our method to registering brain MR images, and conclude that the proposed framework is competent to improve registration performances substantially.     

PCA-based groupwise image registration for quantitative MRI

Quantitative magnetic resonance imaging (qMRI) is a technique for estimating quantitative tissue properties, such as the T₁ and T₂ relaxation times, apparent diffusion coefficient (ADC), and various perfusion measures. This estimation is achieved by acquiring multiple images with different acquisition parameters (or at multiple time points after injection of a contrast agent) and by fitting a qMRI signal model to the image intensities. Image registration is often necessary to compensate for misalignments due to subject motion and/or geometric distortions caused by the acquisition. However, large differences in image appearance make accurate image registration challenging. In this work, we propose a groupwise image registration method for compensating misalignment in qMRI. The groupwise formulation of the method eliminates the requirement of choosing a reference image, thus avoiding a registration bias. The method minimizes a cost function that is based on principal component analysis (PCA), exploiting the fact that intensity changes in qMRI can be described by a low-dimensional signal model, but not requiring knowledge on the specific acquisition model. The method was evaluated on 4D CT data of the lungs, and both real and synthetic images of five different qMRI applications: T₁ mapping in a porcine heart, combined T₁ and T₂ mapping in carotid arteries, ADC mapping in the abdomen, diffusion tensor mapping in the brain, and dynamic contrast-enhanced mapping in the abdomen. Each application is based on a different acquisition model. The method is compared to a mutual information-based pairwise registration method and four other state-of-the-art groupwise registration methods. Registration accuracy is evaluated in terms of the precision of the estimated qMRI parameters, overlap of segmented structures, distance between corresponding landmarks, and smoothness of the deformation. In all qMRI applications the proposed method performed better than or equally well as competing methods, while avoiding the need to choose a reference image. It is also shown that the results of the conventional pairwise approach do depend on the choice of this reference image. We therefore conclude that our groupwise registration method with a similarity measure based on PCA is the preferred technique for compensating misalignments in qMRI.