Reconstruction of a 3D surface from video that is robust to missing data and outliers: application to minimally invasive surgery using stereo and mono endoscopes

Minimally invasive surgery (MIS) offers great benefits to patients compared with open surgery. Nevertheless during MIS surgeons often need to contend with a narrow field-of-view of the endoscope and obstruction from other surgical instruments. He/she may also need to relate the surgical scene to information derived from previously acquired 3D medical imaging. We thus present a new framework to reconstruct the 3D surface of an internal organ from endoscopic images which is robust to measurement noise, missing data and outliers. This can provide 3D surface with a wide field-of-view for surgeons, and it can also be used for 3D-3D registration of the anatomy to pre-operative CT/MRI data for use in image guided interventions. Our proposed method first removes most of the outliers using an outlier removal method that is based on the trilinear constraints over three images. Then data that are missing from one or more of the video images (missing data) and 3D structure are recovered using the structure from motion (SFM) technique. Evolutionary agents are applied to improve both the efficiency of data recovery and robustness to outliers. Furthermore, an incremental bundle adjustment strategy is used to refine the camera parameters and 3D structure and produce a more accurate 3D surface. Experimental results with synthetic data show that the method is able to reconstruct surfaces in the presence of feature tracking errors (up to 5 pixel standard deviation) and a large amount of missing data (up to 50%). Experiments on a realistic phantom model and in vivo data further demonstrate the good performance of the proposed approach in terms of accuracy (1.7 mm residual phantom surface error) and robustness (50% missing data rate, and 20% outliers in in vivo experiments).     

Evaluation of voxel-based registration of 3-D power Doppler ultrasound and 3-D magnetic resonance angiographic images of carotid arteries

Spatial registration and fusion of ultrasound (US) images with other modalities may aid clinical interpretation. We implemented and evaluated on patient data an automated retrospective registration of magnetic resonance angiography (MRA) carotid bifurcation images with 3-D power Doppler ultrasound (PD US) and indirectly with 3-D B-mode US. Volumes were initially thresholded to reduce the uncorrelated noise signals. The registration algorithm subsequently maximized the mutual information measure between the PD US and 3-D MRA via iterative simplex search to find best "rigid body" transformation. We rated the performance of the algorithm visually on (n = 5) clinical MRA and 3-D PD US datasets. We also evaluated quantitatively the effect of thresholding, initial misalignment of the paired volumes and the reproducibility registration. We investigated the effect of image artefacts by simulation experiments. Preregistration misalignments of up to 5 mm in the transaxial plane, up to 10 mm along the axis of the carotids and up to 40 degrees resulted in 107 of 110 successful registrations, with translational and rotational errors of 0.32 mm +/- 0.3 mm and 1.6 +/- 2.1 degrees. The algorithm was not affected by missing arterial segments of up to 8 mm in length. The average registration time was 4 min. We conclude that the algorithm could be applied to 3-D US PD and MRA data for automated multimodality registration of carotid vessels without the use of fiducials.     

Validation of a hybrid Doppler ultrasound vessel-based registration algorithm for neurosurgery

Purpose

We describe and validate a novel hybrid nonlinear vessel registration algorithm for intra-operative updating of preoperative magnetic resonance (MR) images using Doppler ultrasound (US) images acquired on the dura for the correction of brain-shift and registration inaccuracies. We also introduce an US vessel appearance simulator that generates vessel images similar in appearance to that acquired with US from MR angiography data.

Methods

Our registration uses the minimum amount of preprocessing to extract vessels from the raw volumetric images. This prevents the removal of important registration information and minimizes the introduction of artifacts that may affect robustness, while reducing the amount of extraneous information in the image to be processed, thus improving the convergence speed of the algorithm. We then completed 3 rounds of validation for our vessel registration method for robustness and accuracy using (i) a large number of synthetic trials generated with our US vessel simulator, (ii) US images acquired from a real physical phantom made from polyvinyl alcohol cryogel, and (iii) real clinical data gathered intra-operatively from 3 patients.

Results

Resulting target registration errors (TRE) of less than 2.5?mm are achieved in more than 90?% of the synthetic trials when the initial TREs are less than 20?mm. TREs of less than 2?mm were achieved when the technique was applied to the physical phantom, and TREs of less than 3?mm were achieved on clinical data.

Conclusions

These test trials show that the proposed algorithm is not only accurate but also highly robust to noise and missing vessel segments when working with US images acquired in a wide range of real-world conditions.     

Integrating watersheds and critical point analysis for object detection in discrete 2D images

This paper presents an improved method for the detection of "significant" low-level objects in medical images. The method overcomes topological problems where multiple redundant saddle points are detected in digital images. Information derived from watershed regions is used to select and refine saddle points in the discrete domain and to construct the watersheds and watercourses (ridges and valleys). We also demonstrate an improved method of pruning the tessellation by which to define low level objects in zero order images. The algorithm was applied on a set of medical images with promising results. Evaluation was based on theoretical analysis and human observer experiments.     

Geodesic density regression for correcting 4DCT pulmonary respiratory motion artifacts

Pulmonary respiratory motion artifacts are common in four-dimensional computed tomography (4DCT) of lungs and are caused by missing, duplicated, and misaligned image data. This paper presents a geodesic density regression (GDR) algorithm to correct motion artifacts in 4DCT by correcting artifacts in one breathing phase with artifact-free data from corresponding regions of other breathing phases. The GDR algorithm estimates an artifact-free lung template image and a smooth, dense, 4D (space plus time) vector field that deforms the template image to each breathing phase to produce an artifact-free 4DCT scan. Correspondences are estimated by accounting for the local tissue density change associated with air entering and leaving the lungs, and using binary artifact masks to exclude regions with artifacts from image regression. The artifact-free lung template image is generated by mapping the artifact-free regions of each phase volume to a common reference coordinate system using the estimated correspondences and then averaging. This procedure generates a fixed view of the lung with an improved signal-to-noise ratio. The GDR algorithm was evaluated and compared to a state-of-the-art geodesic intensity regression (GIR) algorithm using simulated CT time-series and 4DCT scans with clinically observed motion artifacts. The simulation shows that the GDR algorithm has achieved significantly more accurate Jacobian images and sharper template images, and is less sensitive to data dropout than the GIR algorithm. We also demonstrate that the GDR algorithm is more effective than the GIR algorithm for removing clinically observed motion artifacts in treatment planning 4DCT scans. Our code is freely available at https://github.com/Wei-Shao-Reg/GDR.     

基于改进SUSAN算法的医学图像边缘检测

背景:医学图像的边缘检测是医学图像处理中的一项重要的技术,也是医学图像进一步处理的基础。目的:运用改进的SUSAN算法对医学图像进行边缘检测,取得更丰富的医学图像边缘信息,以便于医学图像的进一步处理。方法:运用Sobel算子对SUSAN算法进行了改进,采用C++语言编程,并在VC++6.0开发平台上实现了改进算法。结果与结论:实验结果表明,该算法能实现阈值的自适应选取,对医学图像中的低对比度的图像边缘有较好的检测效果。     

Eigenrank by committee: Von-Neumann entropy based data subset selection and failure prediction for deep learning based medical image segmentation

Manual delineation of anatomy on existing images is the basis of developing deep learning algorithms for medical image segmentation. However, manual segmentation is tedious. It is also expensive because clinician effort is necessary to ensure correctness of delineation. Consequently most algorithm development is based on a tiny fraction of the vast amount of imaging data collected at a medical center. Thus, selection of a subset of images from hospital databases for manual delineation - so that algorithms trained on such data are accurate and tolerant to variation, becomes an important challenge. We address this challenge using a novel algorithm. The proposed algorithm named ‘Eigenrank by Committee’ (EBC) first computes the degree of disagreement between segmentations generated by each DL model in a committee. Then, it iteratively adds to the committee, a DL model trained on cases where the disagreement is maximal. The disagreement between segmentations is quantified by the maximum eigenvalue of a Dice coefficient disagreement matrix a measure closely related to the Von Neumann entropy. We use EBC for selecting data subsets for manual labeling from a larger database of spinal canal segmentations as well as intervertebral disk segmentations. U-Nets trained on these subsets are used to generate segmentations on the remaining data. Similar sized data subsets are also randomly sampled from the respective databases, and U-Nets are trained on these random subsets as well. We found that U-Nets trained using data subsets selected by EBC, generate segmentations with higher average Dice coefficients on the rest of the database than U-Nets trained using random sampling (p < 0.05 using t-tests comparing averages). Furthermore, U-Nets trained using data subsets selected by EBC generate segmentations with a distribution of Dice coefficients that demonstrate significantly (p < 0.05 using Bartlett’s test) lower variance in comparison to U-Nets trained using random sampling for all datasets. We believe that this lower variance indicates that U-Nets trained with EBC are more robust than U-Nets trained with random sampling.     

Availability of Pediatric Emergency Visit Data from Existing Data Sources

Objectives: To determine the availability and completeness of selected data elements from administrative and clinical sources for emergency department (ED) visits in a national pediatric research network. Methods: This was a retrospective study of 25 EDs in the Pediatric Emergency Care Applied Research Network. Data were obtained from two sources at each ED: 1) extant electronic administrative data for all visits during a 12-month period in 2002 and 2) data abstracted from medical records by trained abstractors for visits during ten randomly selected days over a three-month period in 2003. Epidemiologic data were obtained for all visits and additional clinical data for patients with two target conditions: asthma and fractures. Results: A total of 749,036 visits were analyzed from administrative sources and 12,756 medical records abstracted. Data availability varied by element, method of capture, and site. From administrative sources, data on insurance type were the most complete (1.3% overall missing; range, 0%–18.5% for individual sites), whereas mode of arrival (25.5% missing) and triage time (65.3%) were the least complete. Disposition was missing in only 1.2% of medical records overall (range, 0%–5%) and diagnosis was missing in 3% (range, 0%–16%); these were missing from 14.4% and 10.5%, respectively, of administrative sources. Among visits with injury diagnoses, E-codes were missing in 27% of cases. For patients with asthma ( n = 861), documentation of specific elements of the clinical examination by nurses and physicians was also variable. Conclusions: Data elements important in emergency medical care for children are frequently missing in existing administrative and medical record sources; completeness varies widely across EDs. Researchers must be aware of these limitations in the use of existing data when planning studies.     

Multi-task learning for the segmentation of organs at risk with label dependence

Automatic segmentation of organs at risk is crucial to aid diagnoses and remains a challenging task in medical image analysis domain. To perform the segmentation, we use multi-task learning (MTL) to accurately determine the contour of organs at risk in CT images. We train an encoder-decoder network for two tasks in parallel. The main task is the segmentation of organs, entailing a pixel-level classification in the CT images, and the auxiliary task is the multi-label classification of organs, entailing an image-level multi-label classification of the CT images. To boost the performance of the multi-label classification, we propose a weighted mean cross entropy loss function for the network training, where the weights are the global conditional probability between two organs. Based on MTL, we optimize the false positive filtering (FPF) algorithm to decrease the number of falsely segmented organ pixels in the CT images. Specifically, we propose a dynamic threshold selection (DTS) strategy to prevent true positive rates from decreasing when using the FPF algorithm. We validate these methods on the public ISBI 2019 segmentation of thoracic organs at risk (SegTHOR) challenge dataset and a private medical organ dataset. The experimental results show that networks using our proposed methods outperform basic encoder-decoder networks without increasing the training time complexity.     

机器学习XGBoost算法在医学领域的应用研究进展

机器学习XGBoost算法于2014年提出,其基于boosting算法展开,在许多数据科学大赛上都显示出了极高的可用性和优异性能。目前基于XGBoost算法构建的分类或回归预测模型已经广泛地运用于医疗保健、金融、教育、制造等领域的数据分析中。在医药学领域中XGBoost已广泛应用于疾病诊断以及疾病发生风险、转归与预后、合理安全用药和药物研发的等方面,并且在这些领域中提供了具有极大可能性的解决方案,有助于提高决策的效率和质量,降低假阳性率。同时,XGBoost算法在处理数据缺失值时,能自动学习分裂方向;在处理大型数据集时,能够模拟非线性效应,具有较高的效率和准确性。      

Algorithm for haptic rendering of reconstructed 3D solid organs

Purpose  The use of haptic (the sensing of touch) technology as an interactive tool for new diagnostic procedures is an important and interesting goal because of the potential benefits. Materials and methods  We developed an algorithm for integration of haptic sensing in a medical 3D visualization environment. 3D reconstructions were generated from a stack of medical preoperative images. The innovation of the presented work is the improvement on the behavior of the haptic rendering over previous algorithms. First, the tool we developed for 3D reconstruction is presented. The classical pipeline for surface 3D reconstruction is reviewed from a parametric point of view. These parameters play an important role in the analysis of the haptic behavior. In addition, all the parameters of the reconstruction are accessible and can be modified on-line during the reconstruction procedure. Next, the software architecture used for the integration of the haptic devices is described. The haptic rendering algorithm is detailed, including the collision detection algorithm (a simple ray-tracing scheme programmed using VTK) that is used with the medical images. Results  The results obtained by evaluation of the haptic algorithm’s behavior are presented, demonstrating acceptance of the interactive tool by medical professionals. Conclusion  An improved method for haptic sensing and interaction in a 3D medical visualization environment is feasible and promises to improve image-guided interventions and minimally invasive surgery.     

Automatic elastic image registration by interpolation of 3D rotations and translations from discrete rigid-body transformations

We present an algorithm for automatic elastic registration of three-dimensional (3D) medical images. Our algorithm initially recovers the global spatial mismatch between the reference and floating images, followed by hierarchical octree-based subdivision of the reference image and independent registration of the floating image with the individual subvolumes of the reference image at each hierarchical level. Global as well as local registrations use the six-parameter full rigid-body transformation model and are based on maximization of normalized mutual information (NMI). To ensure robustness of the subvolume registration with low voxel counts, we calculate NMI using a combination of current and prior mutual histograms. To generate a smooth deformation field, we perform direct interpolation of six-parameter rigid-body subvolume transformations obtained at the last subdivision level. Our interpolation scheme involves scalar interpolation of the 3D translations and quaternion interpolation of the 3D rotational pose. We analyzed the performance of our algorithm through experiments involving registration of synthetically deformed computed tomography (CT) images. Our algorithm is general and can be applied to image pairs of any two modalities of most organs. We have demonstrated successful registration of clinical whole-body CT and positron emission tomography (PET) images using this algorithm. The registration accuracy for this application was evaluated, based on validation using expert-identified anatomical landmarks in 15 CT-PET image pairs. The algorithm's performance was comparable to the average accuracy observed for three expert-determined registrations in the same 15 image pairs.     

计算机辅助医学图像三维重建的算法分析

背景：三维重建技术是采用计算机技术对二维医学图像进行边界识别,重新还原出被检组织或器官的三维图像。目的：分忻在不同情况下进行医学图像三维重建时如何进行算法的选择。。方法：采用计算机检索中国期刊全文数据库和Pubmed数据库。中文检索词为“医学图像,三维重建,面绘制,体绘制”,英文检索词为“medicalimages,three—dimensionalreconstruction,surfacerendering,volumerendering”。检索与医学图像三维重建算法相关的文献33篇,从面绘制重置方法和体绘制重置方法的实现原理、实现复杂度、实时显示情况等方面进行分析。结果与结论：目前,医学图像三维重建根据绘制过程中数据描述方法的不同可分为三大类：面绘制方法、体绘制方法和混合绘制方法。通过对面绘制和体绘制方法中不同算法的分析,可以看到面绘制方法在算法效率和实时交互性上是优于体绘制的,虽然面绘制方法在绘制时候会丢失许多细节,使得绘制图像效果不理想,但是由于其算法比较简单,占用内存资源少,所以目前得到了广泛的运用。体绘制方法是对体数据场中的体索进行直接操作,可以绘制出三维数据场中更丰富的信息,因此体绘制方法的绘制效果优于面绘制方法。     

Cross contrast multi-channel image registration using image synthesis for MR brain images

Multi-modal deformable registration is important for many medical image analysis tasks such as atlas alignment, image fusion, and distortion correction. Whereas a conventional method would register images with different modalities using modality independent features or information theoretic metrics such as mutual information, this paper presents a new framework that addresses the problem using a two-channel registration algorithm capable of using mono-modal similarity measures such as sum of squared differences or cross-correlation. To make it possible to use these same-modality measures, image synthesis is used to create proxy images for the opposite modality as well as intensity-normalized images from each of the two available images. The new deformable registration framework was evaluated by performing intra-subject deformation recovery, intra-subject boundary alignment, and inter-subject label transfer experiments using multi-contrast magnetic resonance brain imaging data. Three different multi-channel registration algorithms were evaluated, revealing that the framework is robust to the multi-channel deformable registration algorithm that is used. With a single exception, all results demonstrated improvements when compared against single channel registrations using the same algorithm with mutual information.     

Neighbourhood approximation using randomized forests

Leveraging available annotated data is an essential component of many modern methods for medical image analysis. In particular, approaches making use of the “neighbourhood” structure between images for this purpose have shown significant potential. Such techniques achieve high accuracy in analysing an image by propagating information from its immediate “neighbours” within an annotated database. Despite their success in certain applications, wide use of these methods is limited due to the challenging task of determining the neighbours for an out-of-sample image. This task is either computationally expensive due to large database sizes and costly distance evaluations, or infeasible due to distance definitions over semantic information, such as ground truth annotations, which is not available for out-of-sample images.This article introduces Neighbourhood Approximation Forests (NAFs), a supervised learning algorithm providing a general and efficient approach for the task of approximate nearest neighbour retrieval for arbitrary distances. Starting from an image training database and a user-defined distance between images, the algorithm learns to use appearance-based features to cluster images approximating the neighbourhood structured induced by the distance. NAF is able to efficiently infer nearest neighbours of an out-of-sample image, even when the original distance is based on semantic information. We perform experimental evaluation in two different scenarios: (i) age prediction from brain MRI and (ii) patch-based segmentation of unregistered, arbitrary field of view CT images. The results demonstrate the performance, computational benefits, and potential of NAF for different image analysis applications.     

MAP-based kinetic analysis for voxel-by-voxel compartment model estimation: detailed imaging of the cerebral glucose metabolism using FDG

We propose a novel algorithm for voxel-by-voxel compartment model analysis based on a maximum a posteriori (MAP) algorithm. Voxel-by-voxel compartment model analysis can derive functional images of living tissues, but it suffers from high noise statistics in voxel-based PET data and extended calculation times. We initially set up a feature space of the target radiopharmaceutical composed of a measured plasma time activity curve and a set of compartment model parameters, and measured the noise distribution of the PET data. The dynamic PET data were projected onto the feature space, and then clustered using the Mahalanobis distance. Our method was validated using simulation studies, and compared with ROI-based ordinary kinetic analysis for FDG. The parametric images exhibited an acceptable linear relation with the simulations and the ROI-based results, and the calculation time took about 10 min. We therefore concluded that our proposed MAP-based algorithm is practical.     

三维多模态医学图像自动配准方法研究

提出一种综合应用图像分割与互信息的医学图像自动配准方法.首先采用门限法和数学形态学方法进行预处理,再用k-means方法进行分割,之后采用基于互信息的Powell优化方法配准.将该方法用于磁共振图像(MRI)和正电子发射断层扫描(PET)临床医学图像配准,得到较满意的效果.     

Efficient computational fluid dynamics mesh generation by image registration

Most implementations of computational fluid dynamics (CFD) solutions require a discretisation or meshing of the solution domain. The production from a medical image of a computationally efficient mesh representing the structures of interest can be time consuming and labour-intensive, and remains a major bottleneck in the clinical application of CFD. This paper presents a method for deriving a patient-specific mesh from a medical image. The method uses volumetric registration of a pseudo-image, produced from an idealised template mesh, with the medical image. The registration algorithm used is robust and computationally efficient. The accuracy of the new algorithm is measured in terms of the distance between a registered surface and a known surface, for image data derived from casts of the lumen of two different vessels. The true surface is identified by laser profiling. The average distance between the surface points measured by the laser profiler and the surface of the mapped mesh is better than 0.2 mm. For the images analysed, the new algorithm is shown to be 2-3 times more accurate than a standard published algorithm based on maximising normalised mutual information. Computation times are approximately 18 times faster for the new algorithm than the standard algorithm. Examples of the use of the algorithm on two clinical examples are also given. The registration methodology lends itself immediately to the construction of dynamic mesh models in which vessel wall motion is obtained directly using registration.