A method of image registration for small animal, multi-modality imaging

Many research institutions have a full suite of preclinical tomographic scanners to answer biomedical questions in vivo. Routine multi-modality imaging requires robust registration of images generated by various tomographs. We have implemented a hardware registration method for preclinical imaging that is similar to that used in the combined positron emission tomography (PET)/computed tomography (CT) scanners in the clinic. We designed an imaging chamber which can be rigidly and reproducibly mounted on separate microPET and microCT scanners. We have also designed a three-dimensional grid phantom with 1288 lines that is used to generate the spatial transformation matrix from software registration using a 15-parameter perspective model. The imaging chamber works in combination with the registration phantom synergistically to achieve the image registration goal. We verified that the average registration error between two imaging modalities is 0.335 mm using an in vivo mouse bone scan. This paper also estimates the impact of image misalignment on PET quantitation using attenuation corrections generated from misregistered images. Our technique is expected to produce PET quantitation errors of less than 5%. The methods presented are robust and appropriate for routine use in high throughput animal imaging facilities.     

Digimouse: a 3D whole body mouse atlas from CT and cryosection data

We have constructed a three-dimensional (3D) whole body mouse atlas from coregistered x-ray CT and cryosection data of a normal nude male mouse. High quality PET, x-ray CT and cryosection images were acquired post mortem from a single mouse placed in a stereotactic frame with fiducial markers visible in all three modalities. The image data were coregistered to a common coordinate system using the fiducials and resampled to an isotropic 0.1 mm voxel size. Using interactive editing tools we segmented and labelled whole brain, cerebrum, cerebellum, olfactory bulbs, striatum, medulla, masseter muscles, eyes, lachrymal glands, heart, lungs, liver, stomach, spleen, pancreas, adrenal glands, kidneys, testes, bladder, skeleton and skin surface. The final atlas consists of the 3D volume, in which the voxels are labelled to define the anatomical structures listed above, with coregistered PET, x-ray CT and cryosection images. To illustrate use of the atlas we include simulations of 3D bioluminescence and PET image reconstruction. Optical scatter and absorption values are assigned to each organ to simulate realistic photon transport within the animal for bioluminescence imaging. Similarly, 511 keV photon attenuation values are assigned to each structure in the atlas to simulate realistic photon attenuation in PET. The Digimouse atlas and data are available at http://neuroimage.usc.edu/Digimouse.html.     

Simultaneous molecular and anatomical imaging of the mouse in vivo

Non-invasive imaging technologies are opening up new windows into mouse biology. We have developed a mouse imaging system that integrates positron emission tomography (PET) with x-ray computed tomography (CT), allowing simultaneous anatomic and molecular imaging in vivo with the potential for precise registration of the two image volumes. The x-ray system consists of a compact mini-focal x-ray tube and an amorphous selenium flat panel x-ray detector with a low-noise CMOS readout. The PET system uses planar arrays of lutetium oxyorthosilicate scintillator coupled to position-sensitive photomultiplier tubes. We describe the design of this dual-modality imaging system and show, for the first time, simultaneously acquired PET and CT images in a phantom and in mice.     

Standard atlas space for C57BL/6J neonatal mouse brain

A standard atlas space with stereotaxic co-ordinates for the postnatal day 0 (P0) C57BL/6J mouse brain was constructed from the average of eight individual co-registered MR image volumes. Accuracy of registration and morphometric variations in structures between subjects were analyzed statistically. We also applied this atlas coordinate system to data acquired using different imaging protocols as well as to a high-resolution histological atlas obtained from separate animals. Mapping accuracy in the atlas space was examined to determine the applicability of this atlas framework. The results show that the atlas space defined here provides a stable framework for image registration for P0 normal mouse brains. With an appropriate feature-based co-registration strategy, the probability atlas can also provide an accurate anatomical map for images acquired using invasive imaging methods. The atlas templates and the probability map of the anatomical labels are available at .     

Retrospective Evaluation of PET-MRI Registration Algorithms

The purpose of this study is to evaluate the accuracy of registration positron emission tomography (PET) head images to the MRI-based brain atlas. The [¹⁸F]fluoro-2-deoxyglucose PET images were normalized to the MRI-based brain atlas using nine registration algorithms including objective functions of ratio image uniformity (RIU), normalized mutual information (NMI), and normalized cross correlation (CC) and transformation models of rigid-body, linear, affine, and nonlinear transformations. The accuracy of normalization was evaluated by visual inspection and quantified by the gray matter (GM) concordance between normalized PET images and the brain atlas. The linear and affine registration based on the RIU provided the best GM concordance (average similarity index of 0.71 for both). We also observed that the GM concordances of linear and affine registration were higher than those of the rigid and nonlinear registration among the methods evaluated.     

Registration of 2D x-ray images to 3D MRI by generating pseudo-CT data

Spatial and soft tissue information provided by magnetic resonance imaging can be very valuable during image-guided procedures, where usually only real-time two-dimensional (2D) x-ray images are available. Registration of 2D x-ray images to three-dimensional (3D) magnetic resonance imaging (MRI) data, acquired prior to the procedure, can provide optimal information to guide the procedure. However, registering x-ray images to MRI data is not a trivial task because of their fundamental difference in tissue contrast. This paper presents a technique that generates pseudo-computed tomography (CT) data from multi-spectral MRI acquisitions which is sufficiently similar to real CT data to enable registration of x-ray to MRI with comparable accuracy as registration of x-ray to CT. The method is based on a k-nearest-neighbors (kNN)-regression strategy which labels voxels of MRI data with CT Hounsfield Units. The regression method uses multi-spectral MRI intensities and intensity gradients as features to discriminate between various tissue types. The efficacy of using pseudo-CT data for registration of x-ray to MRI was tested on ex vivo animal data. 2D-3D registration experiments using CT and pseudo-CT data of multiple subjects were performed with a commonly used 2D-3D registration algorithm. On average, the median target registration error for registration of two x-ray images to MRI data was approximately 1 mm larger than for x-ray to CT registration. The authors have shown that pseudo-CT data generated from multi-spectral MRI facilitate registration of MRI to x-ray images. From the experiments it could be concluded that the accuracy achieved was comparable to that of registering x-ray images to CT data.     

多模态小动物成像设备在恶性肿瘤应用中的研究进展

PET正成为强有力的临床前研究工具,与其他在体成像模式联合可以实现优势互补,在恶性肿瘤基础研究中扮演重要角色。本文综述小动物PET成像系统、小动物PET/解剖双模态成像系统、小动物PET/光学双模态成像系统和小动物PET/解剖成像/光学成像多模态成像系统在恶性肿瘤中的应用情况,并对其发展趋势进行了展望。     

Deformable and rigid registration of MRI and microPET images for photodynamic therapy of cancer in mice

We are investigating imaging techniques to study the tumor response to photodynamic therapy (PDT). Positron emission tomography (PET) can provide physiological and functional information. High-resolution magnetic resonance imaging (MRI) can provide anatomical and morphological changes. Image registration can combine MRI and PET images for improved tumor monitoring. In this study, we acquired high-resolution MRI and microPET 18F-fluorodeoxyglucose (FDG) images from C3H mice with RIF-1 tumors that were treated with Pc 4-based PDT. We developed two registration methods for this application. For registration of the whole mouse body, we used an automatic three-dimensional, normalized mutual information algorithm. For tumor registration, we developed a finite element model (FEM)-based deformable registration scheme. To assess the quality of whole body registration, we performed slice-by-slice review of both image volumes; manually segmented feature organs, such as the left and right kidneys and the bladder, in each slice; and computed the distance between corresponding centroids. Over 40 volume registration experiments were performed with MRI and microPET images. The distance between corresponding centroids of organs was 1.5 +/- 0.4 mm which is about 2 pixels of microPET images. The mean volume overlap ratios for tumors were 94.7% and 86.3% for the deformable and rigid registration methods, respectively. Registration of high-resolution MRI and microPET images combines anatomical and functional information of the tumors and provides a useful tool for evaluating photodynamic therapy.     

Diffuse optical tomography with a priori anatomical information

Diffuse optical tomography (DOT) poses a typical ill-posed inverse problem with a limited number of measurements and inherently low spatial resolution. In this paper, we propose a hierarchical Bayesian approach to improve spatial resolution and quantitative accuracy by using a priori information provided by a secondary high resolution anatomical imaging modality, such as magnetic resonance (MR) or x-ray. In such a dual imaging approach, while the correlation between optical and anatomical images may be high, it is not perfect. For example, a tumour may be present in the optical image, but may not be discernable in the anatomical image. The proposed hierarchical Bayesian approach allows incorporation of partial a priori knowledge about the noise and unknown optical image models, thereby capturing the function-anatomy correlation effectively. We present a computationally efficient iterative algorithm to simultaneously estimate the optical image and the unknown a priori model parameters. Extensive numerical simulations demonstrate that the proposed method avoids undesirable bias towards anatomical prior information and leads to significantly improved spatial resolution and quantitative accuracy.     

Joint model of motion and anatomy for PET image reconstruction

Anatomy-based positron emission tomography (PET) image enhancement techniques have been shown to have the potential for improving PET image quality. However, these techniques assume an accurate alignment between the anatomical and the functional images, which is not always valid when imaging the chest due to respiratory motion. In this article, we present a joint model of both motion and anatomical information by integrating a motion-incorporated PET imaging system model with an anatomy-based maximum a posteriori image reconstruction algorithm. The mismatched anatomical information due to motion can thus be effectively utilized through this joint model. A computer simulation and a phantom study were conducted to assess the efficacy of the joint model, whereby motion and anatomical information were either modeled separately or combined. The reconstructed images in each case were compared to corresponding reference images obtained using a quadratic image prior based maximum a posteriori reconstruction algorithm for quantitative accuracy. Results of these studies indicated that while modeling anatomical information or motion alone improved the PET image quantitation accuracy, a larger improvement in accuracy was achieved when using the joint model. In the computer simulation study and using similar image noise levels, the improvement in quantitation accuracy compared to the reference images was 5.3% and 19.8% when using anatomical or motion information alone, respectively, and 35.5% when using the joint model. In the phantom study, these results were 5.6%, 5.8%, and 19.8%, respectively. These results suggest that motion compensation is important in order to effectively utilize anatomical information in chest imaging using PET. The joint motion-anatomy model presented in this paper provides a promising solution to this problem.     

DigiWarp: a method for deformable mouse atlas warping to surface topographic data

For pre-clinical bioluminescence or fluorescence optical tomography, the animal's surface topography and internal anatomy need to be estimated for improving the quantitative accuracy of reconstructed images. The animal's surface profile can be measured by all-optical systems, but estimation of the internal anatomy using optical techniques is non-trivial. A 3D anatomical mouse atlas may be warped to the estimated surface. However, fitting an atlas to surface topography data is challenging because of variations in the posture and morphology of imaged mice. In addition, acquisition of partial data (for example, from limited views or with limited sampling) can make the warping problem ill-conditioned. Here, we present a method for fitting a deformable mouse atlas to surface topographic range data acquired by an optical system. As an initialization procedure, we match the posture of the atlas to the posture of the mouse being imaged using landmark constraints. The asymmetric L(2) pseudo-distance between the atlas surface and the mouse surface is then minimized in order to register two data sets. A Laplacian prior is used to ensure smoothness of the surface warping field. Once the atlas surface is normalized to match the range data, the internal anatomy is transformed using elastic energy minimization. We present results from performance evaluation studies of our method where we have measured the volumetric overlap between the internal organs delineated directly from MRI or CT and those estimated by our proposed warping scheme. Computed Dice coefficients indicate excellent overlap in the brain and the heart, with fair agreement in the kidneys and the bladder.     

Clinically feasible reconstruction of 3D whole-body PET/CT data using blurred anatomical labels

We present the results of utilizing aligned anatomical information from CT images to locally adjust image smoothness during the reconstruction of three-dimensional (3D) whole-body positron emission tomography (PET) data. The ability of whole-body PET imaging to detect malignant neoplasms is becoming widely recognized. Potentially useful, however, is the role of whole-body PET in quantitative estimation of tracer uptake. The utility of PET in oncology is often limited by the high level of statistical noise in the images. Reduction in noise can be obtained by incorporating a priori image smoothness information from correlated anatomical information during the reconstruction of PET data. A combined PET/CT scanner allows the acquisition of accurately aligned PET and x-ray CT whole-body data. We use the Fourier rebinning algorithm (FORE) to accurately convert the 3D PET data to two-dimensional (2D) data to accelerate the image reconstruction process. The 2D datasets are reconstructed with successive over-relaxation of a penalized weighted least squares (PWLS) objective function to model the statistics of the acquisition, data corrections, and rebinning. A 3D voxel label model is presented that incorporates the anatomical information via the penalty weights of the PWLS objective function. This combination of FORE + PWLS + labels was developed as it allows for both reconstruction of 3D whole-body data sets in clinically feasible times and also the inclusion of anatomical information in such a way that convergence can be guaranteed. Since mismatches between anatomical (CT) and functional (PET) data are unavoidable in practice, the labels are 'blurred' to reflect the uncertainty associated with the anatomical information. Simulated and experimental results show the potential advantage of incorporating anatomical information by using blurred labels to calculate the penalty weights. We conclude that while the effect of this method on detection tasks is complicated and unclear, there is an improvement on the estimation task.     

Incorporation of Preprocedural PET into CT-Guided Radiofrequency Ablation of Hepatic Metastases: a Nonrigid Image Registration Validation Study

This study evaluates the accuracy of augmenting initial intraprocedural computed tomography (CT) during radiofrequency ablation (RFA) of hepatic metastases with preprocedural positron emission tomography (PET) through a hardware-accelerated implementation of an automatic nonrigid PET–CT registration algorithm. The feasibility of augmenting intraprocedural CT with preprocedural PET to improve localization of CT-invisible but PET-positive tumors with images from actual RFA was explored. Preprocedural PET and intraprocedural CT images from 18 cases of hepatic RFA were included. All PET images in the study originated from a hybrid PET/CT scanner, and PET–CT registration was performed in two ways: (1) direct registration of preprocedural PET with intraprocedural CT and (2) indirect registration of preprocedural CT (i.e., the CT of hybrid PET/CT scan) with intraprocedural CT. A hardware-accelerated registration took approximately 2 min. Calculated registration errors were 7.0 and 8.4 mm for the direct and indirect methods, respectively. Overall, the direct registration was found to be statistically not distinct from that performed by a group of clinical experts. The accuracy, execution speed, and compactness of our implementation of nonrigid image registration suggest that existing PET can be overlaid on intraprocedural CT, promising a novel, technically feasible, and clinically viable approach for PET augmentation of CT guidance of RFA.     

Phantom validation of coregistration of PET and CT for image-guided radiotherapy

Radiotherapy treatment planning integrating positron emission tomography (PET) and computerized tomography (CT) is rapidly gaining acceptance in the clinical setting. Although hybrid systems are available, often the planning CT is acquired on a dedicated system separate from the PET scanner. A limiting factor to using PET data becomes the accuracy of the CT/PET registration. In this work, we use phantom and patient validation to demonstrate a general method for assessing the accuracy of CT/PET image registration and apply it to two multi-modality image registration programs. An IAEA (International Atomic Energy Association) brain phantom and an anthropomorphic head phantom were used. Internal volumes and externally mounted fiducial markers were filled with CT contrast and 18F-fluorodeoxyglucose (FDG). CT, PET emission, and PET transmission images were acquired and registered using two different image registration algorithms. CT/PET Fusion (GE Medical Systems, Milwaukee, WI) is commercially available and uses a semi-automated initial step followed by manual adjustment. Automatic Mutual Information-based Registration (AMIR), developed at our institution, is fully automated and exhibits no variation between repeated registrations. Registration was performed using distinct phantom structures; assessment of accuracy was determined from registration of the calculated centroids of a set of fiducial markers. By comparing structure-based registration with fiducial-based registration, target registration error (TRE) was computed at each point in a three-dimensional (3D) grid that spans the image volume. Identical methods were also applied to patient data to assess CT/PET registration accuracy. Accuracy was calculated as the mean with standard deviation of the TRE for every point in the 3D grid. Overall TRE values for the IAEA brain phantom are: CT/PET Fusion = 1.71 +/- 0.62 mm, AMIR = 1.13 +/- 0.53 mm; overall TRE values for the anthropomorphic head phantom are: CT/PET Fusion = 1.66 +/- 0.53 mm, AMIR = 1.15 +/- 0.48 mm. Precision (repeatability by a single user) measured for CT/PET Fusion: IAEA phantom = 1.59 +/- 0.67 mm and anthropomorphic head phantom = 1.63 +/- 0.52 mm. (AMIR has exact precision and so no measurements are necessary.) One sample patient demonstrated the following accuracy results: CT/PET Fusion = 3.89 +/- 1.61 mm, AMIR = 2.86 +/- 0.60 mm. Semi-automatic and automatic image registration methods may be used to facilitate incorporation of PET data into radiotherapy treatment planning in relatively rigid anatomic sites, such as head and neck. The overall accuracies in phantom and patient images are < 2 mm and < 4 mm, respectively, using either registration algorithm. Registration accuracy may decrease, however, as distance from the initial registration points (CT/PET fusion) or center of the image (AMIR) increases. Additional information provided by PET may improve dose coverage to active tumor subregions and hence tumor control. This study shows that the accuracy obtained by image registration with these two methods is well suited for image-guided radiotherapy.     

Wavelet-based resolution recovery using an anatomical prior provides quantitative recovery for human population phantom PET [¹¹C]raclopride data

The objective of this study was to evaluate a resolution recovery (RR) method using a variety of simulated human brain [11C]raclopride positron emission tomography (PET) images. Simulated datasets of 15 numerical human phantoms were processed by a wavelet-based RR method using an anatomical prior. The anatomical prior was in the form of a hybrid segmented atlas, which combined an atlas for anatomical labelling and a PET image for functional labelling of each anatomical structure. We applied RR to both 60 min static and dynamic PET images. Recovery was quantified in 84 regions, comparing the typical 'true' value for the simulation, as obtained in normal subjects, simulated and RR PET images. The radioactivity concentration in the white matter, striatum and other cortical regions was successfully recovered for the 60 min static image of all 15 human phantoms; the dependence of the solution on accurate anatomical information was demonstrated by the difficulty of the technique to retrieve the subthalamic nuclei due to mismatch between the two atlases used for data simulation and recovery. Structural and functional synergy for resolution recovery (SFS-RR) improved quantification in the caudate and putamen, the main regions of interest, from?-30.1% and?-26.2% to?-17.6% and?-15.1%, respectively, for the 60 min static image and from?-51.4% and?-38.3% to?-27.6% and?-20.3% for the binding potential (BP(ND)) image, respectively. The proposed methodology proved effective in the RR of small structures from brain [11C]raclopride PET images. The improvement is consistent across the anatomical variability of a simulated population as long as accurate anatomical segmentations are provided.     

Noninvasive Technique for Monitoring Drug Transport Through the Murine Cochlea using Micro-Computed Tomography

Local delivery of drugs to the inner ear has the potential to treat inner ear disorders including permanent hearing loss or deafness. Current mathematical models describing the pharmacokinetics of drug delivery to the inner ear have been based on large rodent studies with invasive measurements of concentration at few locations within the cochlea. Hence, estimates of clearance and diffusion parameters are based on fitting measured data with limited spatial resolution to a model. To overcome these limitations, we developed a noninvasive imaging technique to monitor and characterize drug delivery inside the mouse cochlea using micro-computed tomography (μCT). To increase the measurement accuracy, we performed a subject-atlas image registration to exploit the information readily available in the atlas image of the mouse cochlea and pass segmentation or labeling information from the atlas to our μCT scans. The approach presented here has the potential to quantify concentrations at any point along fluid-filled scalae of the inner ear. This may permit determination of spatially dependent diffusion and clearance parameters for enhanced models.     

Temporal subtraction of dual-energy chest radiographs

Temporal subtraction and dual-energy imaging are two enhanced radiography techniques that are receiving increased attention in chest radiography. Temporal subtraction is an image processing technique that facilitates the visualization of pathologic change across serial chest radiographic images acquired from the same patient; dual-energy imaging exploits the differential relative attenuation of x-ray photons exhibited by soft-tissue and bony structures at different x-ray energies to generate a pair of images that accentuate those structures. Although temporal subtraction images provide a powerful mechanism for enhancing visualization of subtle change, misregistration artifacts in these images can mimic or obscure abnormalities. The purpose of this study was to evaluate whether dual-energy imaging could improve the quality of temporal subtraction images. Temporal subtraction images were generated from 100 pairs of temporally sequential standard radiographic chest images and from the corresponding 100 pairs of dual-energy, soft-tissue radiographic images. The registration accuracy demonstrated in the resulting temporal subtraction images was evaluated subjectively by two radiologists. The registration accuracy of the soft-tissue-based temporal subtraction images was rated superior to that of the conventional temporal subtraction images. Registration accuracy also was evaluated objectively through an automated method, which achieved an area-under-the-ROC-curve value of 0.92 in the distinction between temporal subtraction images that demonstrated clinically acceptable and clinically unacceptable registration accuracy. By combining dual-energy soft-tissue images with temporal subtraction, misregistration artifacts can be reduced and superior image quality can be obtained.     

PET/CT imaging for treatment verification after proton therapy: a study with plastic phantoms and metallic implants

The feasibility of off-line positron emission tomography/computed tomography (PET/CT) for routine three dimensional in-vivo treatment verification of proton radiation therapy is currently under investigation at Massachusetts General Hospital in Boston. In preparation for clinical trials, phantom experiments were carried out to investigate the sensitivity and accuracy of the method depending on irradiation and imaging parameters. Furthermore, they addressed the feasibility of PET/CT as a robust verification tool in the presence of metallic implants. These produce x-ray CT artifacts and fluence perturbations which may compromise the accuracy of treatment planning algorithms. Spread-out Bragg peak proton fields were delivered to different phantoms consisting of polymethylmethacrylate (PMMA), PMMA stacked with lung and bone equivalent materials, and PMMA with titanium rods to mimic implants in patients. PET data were acquired in list mode starting within 20 min after irradiation at a commercial luthetium-oxyorthosilicate (LSO)-based PET/CT scanner. The amount and spatial distribution of the measured activity could be well reproduced by calculations based on the GEANT4 and FLUKA Monte Carlo codes. This phantom study supports the potential of millimeter accuracy for range monitoring and lateral field position verification even after low therapeutic dose exposures of 2 Gy, despite the delay between irradiation and imaging. It also indicates the value of PET for treatment verification in the presence of metallic implants, demonstrating a higher sensitivity to fluence perturbations in comparison to a commercial analytical treatment planning system. Finally, it addresses the suitability of LSO-based PET detectors for hadron therapy monitoring. This unconventional application of PET involves countrates which are orders of magnitude lower than in diagnostic tracer imaging, i.e., the signal of interest is comparable to the noise originating from the intrinsic radioactivity of the detector itself. In addition to PET alone, PET/CT imaging provides accurate information on the position of the imaged object and may assess possible anatomical changes during fractionated radiotherapy in clinical applications.     

PET/CT image registration: preliminary tests for its application to clinical dosimetry in radiotherapy

The quality of dosimetry in radiotherapy treatment requires the accurate delimitation of the gross tumor volume. This can be achieved by complementing the anatomical detail provided by CT images through fusion with other imaging modalities that provide additional metabolic and physiological information. Therefore, use of multiple imaging modalities for radiotherapy treatment planning requires an accurate image registration method. This work describes tests carried out on a Discovery LS positron emission/computed tomography (PET/CT) system by General Electric Medical Systems (GEMS), for its later use to obtain images to delimit the target in radiotherapy treatment. Several phantoms have been used to verify image correlation, in combination with fiducial markers, which were used as a system of external landmarks. We analyzed the geometrical accuracy of two different fusion methods with the images obtained with these phantoms. We first studied the fusion method used by the PET/CT system by GEMS (hardware fusion) on the basis that there is satisfactory coincidence between the reconstruction centers in CT and PET systems; and secondly the fiducial fusion, a registration method, by means of least-squares fitting algorithm of a landmark points system. The study concluded with the verification of the centroid position of some phantom components in both imaging modalities. Centroids were estimated through a calculation similar to center-of-mass, weighted by the value of the CT number and the uptake intensity in PET. The mean deviations found for the hardware fusion method were: deltax/ +/-sigma = 3.3 mm +/- 1.0 mm and /deltax/ +/-sigma = 3.6 mm +/- 1.0 mm. These values were substantially improved upon applying fiducial fusion based on external landmark points: /deltax/ +/-sigma = 0.7 mm +/- 0.8 mm and /deltax/ +/-sigma = 0.3 mm 1.7 mm. We also noted that differences found for each of the fusion methods were similar for both the axial and helical CT image acquisition protocols.     

Non-rigid registration between 3D ultrasound and CT images of the liver based on intensity and gradient information

In order to utilize both ultrasound (US) and computed tomography (CT) images of the liver concurrently for medical applications such as diagnosis and image-guided intervention, non-rigid registration between these two types of images is an essential step, as local deformation between US and CT images exists due to the different respiratory phases involved and due to the probe pressure that occurs in US imaging. This paper introduces a voxel-based non-rigid registration algorithm between the 3D B-mode US and CT images of the liver. In the proposed algorithm, to improve the registration accuracy, we utilize the surface information of the liver and gallbladder in addition to the information of the vessels inside the liver. For an effective correlation between US and CT images, we treat those anatomical regions separately according to their characteristics in US and CT images. Based on a novel objective function using a 3D joint histogram of the intensity and gradient information, vessel-based non-rigid registration is followed by surface-based non-rigid registration in sequence, which improves the registration accuracy. The proposed algorithm is tested for ten clinical datasets and quantitative evaluations are conducted. Experimental results show that the registration error between anatomical features of US and CT images is less than 2 mm on average, even with local deformation due to different respiratory phases and probe pressure. In addition, the lesion registration error is less than 3 mm on average with a maximum of 4.5 mm that is considered acceptable for clinical applications.