Skin3D: Detection and longitudinal tracking of pigmented skin lesions in 3D total-body textured meshes

We present an automated approach to detect and longitudinally track skin lesions on 3D total-body skin surface scans. The acquired 3D mesh of the subject is unwrapped to a 2D texture image, where a trained objected detection model, Faster R-CNN, localizes the lesions within the 2D domain. These detected skin lesions are mapped back to the 3D surface of the subject and, for subjects imaged multiple times, we construct a graph-based matching procedure to longitudinally track lesions that considers the anatomical correspondences among pairs of meshes and the geodesic proximity of corresponding lesions and the inter-lesion geodesic distances.We evaluated the proposed approach using 3DBodyTex, a publicly available dataset composed of 3D scans imaging the coloured skin (textured meshes) of 200 human subjects. We manually annotated locations that appeared to the human eye to contain a pigmented skin lesion as well as tracked a subset of lesions occurring on the same subject imaged in different poses. Our results, when compared to three human annotators, suggest that the trained Faster R-CNN detects lesions at a similar performance level as the human annotators. Our lesion tracking algorithm achieves an average matching accuracy of 88% on a set of detected corresponding pairs of prominent lesions of subjects imaged in different poses, and an average longitudinal accuracy of 71% when encompassing additional errors due to lesion detection. As there currently is no other large-scale publicly available dataset of 3D total-body skin lesions, we publicly release over 25,000 3DBodyTex manual annotations, which we hope will further research on total-body skin lesion analysis.     

Deep learning solution for medical image localization and orientation detection

Magnetic Resonance (MR) imaging plays an important role in medical diagnosis and biomedical research. Due to the high in-slice resolution and low through-slice resolution nature of MR imaging, the usefulness of the reconstruction highly depends on the positioning of the slice group. Traditional clinical workflow relies on time-consuming manual adjustment that cannot be easily reproduced. Automation of this task can therefore bring important benefits in terms of accuracy, speed and reproducibility. Current auto-slice-positioning methods rely on automatically detected landmarks to derive the positioning, and previous studies suggest that a large, redundant set of landmarks are required to achieve robust results. However, a costly data curation procedure is needed to generate training labels for those landmarks, and the results can still be highly sensitive to landmark detection errors. More importantly, a set of anatomical landmark locations are not naturally produced during the standard clinical workflow, which makes online learning impossible. To address these limitations, we propose a novel framework for auto-slice-positioning that focuses on localizing the canonical planes within a 3D volume. The proposed framework consists of two major steps. A multi-resolution region proposal network is first used to extract a volume-of-interest, after which a V-net-like segmentation network is applied to segment the orientation planes. Importantly, our algorithm also includes a Performance Measurement Index as an indication of the algorithm’s confidence. We evaluate the proposed framework on both knee and shoulder MR scans. Our method outperforms state-of-the-art automatic positioning algorithms in terms of accuracy and robustness.     

Left ventricle wall motion quantification from echocardiographic images by non-rigid image registration

Purpose

The aim of this study is to evaluate the efficiency of applying a new non-rigid image registration method on two-dimensional echocardiographic images for computing the left ventricle (LV) myocardial motion field over a cardiac cycle.

Methods

The key feature of our method is to register all images in the sequence to a reference image (end-diastole image) using a hierarchical transformation model, which is a combination of an affine transformation for modeling the global LV motion and a free-form deformation (FFD) transformation based on B-splines for modeling the local LV deformation. Registration is done by minimizing a cost function associated with the image similarity based on a global pixel-based matching and the smoothness of transformation. The algorithm uses a fast and robust optimization strategy using a multiresolution approach for the estimation of parameters of the deformation model. The proposed algorithm is evaluated for calculating the displacement curves of two expert-identified anatomical landmarks in apical views of the LV for 10 healthy volunteers and 14 subjects with pathology. The proposed algorithm is also evaluated for classifying the regional LV wall motion abnormality using the calculation of the strain value at the end of systole in 288 segments as scored by two consensual experienced echocardiographers in a three-point scale: 1: normokinesia, 2: hypokinesia, and 3: akinesia. Moreover, we compared the results of the proposed registration algorithm to those previously obtained using the other image registration methods.

Results

Regarding to the reference two experienced echocardiographers, the results demonstrate the proposed algorithm more accurately estimates the displacement curve of the two anatomical landmarks in apical views than the other registration methods in all data set. Moreover, the p values of the t test for the strain value of each segment at the end of systole measured by the proposed algorithm show higher differences than the other registration method. These differences are between each pair of scores in all segments and in three segments of septum independently.

Conclusions

The clinical results show that the proposed algorithm can improve both the calculation of the displacement curve of every point of LV during a cardiac cycle and the classification of regional LV wall motion abnormality. Therefore, this diagnostic system can be used as a useful tool for clinical evaluation of the regional LV function.     

Landmark detection in the chest and registration of lung surfaces with an application to nodule registration

We developed an automated system for registering computed tomography (CT) images of the chest temporally. Our system detects anatomical landmarks, in particular, the trachea, sternum and spine, using an attenuation-based template matching approach. It computes the optimal rigid-body transformation that aligns the corresponding landmarks in two CT scans of the same patient. This transformation then provides an initial registration of the lung surfaces segmented from the two scans. The initial surface alignment is refined step by step in an iterative closest-point (ICP) process. To establish the correspondence of lung surface points, Elias' nearest neighbor algorithm was adopted. Our method improves the processing time of the original ICP algorithm from O(kn log n) to O(kn), where k is the number of iterations and n the number of surface points. The surface transformation is applied to align nodules in the initial CT scan with nodules in the follow-up scan. For 56 out of 58 nodules in the initial CT scans of 10 patients, nodule correspondences in the follow-up scans are established correctly. Our methods can therefore potentially facilitate the radiologist's evaluation of pulmonary nodules on chest CT for interval growth.     

Hubless keypoint-based 3D deformable groupwise registration

We present a novel algorithm for Fast Registration Of image Groups (FROG), applied to large 3D image groups. Our approach extracts 3D SURF keypoints from images, computes matched pairs of keypoints and registers the group by minimizing pair distances in a hubless way i.e. without computing any central mean image. Using keypoints significantly reduces the problem complexity compared to voxel-based approaches, and enables us to provide an in-core global optimization, similar to the Bundle Adjustment for 3D reconstruction. As we aim to register images of different patients, the matching step yields many outliers. Then we propose a new EM-weighting algorithm which efficiently discards outliers. Global optimization is carried out with a fast gradient descent algorithm. This allows our approach to robustly register large datasets. The result is a set of diffeomorphic half transforms which link the volumes together and can be subsequently exploited for computational anatomy and landmark detection. We show experimental results on whole-body CT scans, with groups of up to 103 volumes. On a benchmark based on anatomical landmarks, our algorithm compares favorably with the star-groupwise voxel-based ANTs and NiftyReg approaches while being much faster. We also discuss the limitations of our approach for lower resolution images such as brain MRI.     

The precision of anatomical normalization in the medial temporal lobe using spatial basis functions

We investigated the accuracy of spatial basis function normalization using anatomical landmarks to determine how precisely homologous regions are colocalized. We examined precision in terms of: (1) the number of nonlinear basis functions used by the normalization procedure; (2) the degree of (Bayesian) regularization; and (3) the effect of substituting different templates and how this interacted with the number of basis functions. The face validity of spatial normalization was assessed as a function of these parameters, using the colocalization of homologous landmarks in a test sample of 20 normally developing children and 5 children with bilateral hippocampal pathology. Our results suggest that when optimal normalization parameters are used, anatomical landmarks in the medial temporal lobes are colocalized to within a standard deviation of about 1 mm. When suboptimal parameters are used this standard deviation can increase up to 3 mm. Interestingly the optimal parameters are those that provide a rather constrained normalization as opposed to those that optimize intensity matching at the expense of rendering the warps "unlikely." The implications of our results, for users of voxel-based morphometry, are discussed.     

Serial registration of intraoperative MR images of the brain

The increased use of image-guided surgery systems during neurosurgery has brought to prominence the inaccuracies of conventional intraoperative navigation systems caused by shape changes such as those due to brain shift. We propose a method to track the deformation of the brain and update preoperative images using intraoperative MR images acquired at different crucial time points during surgery. We use a deformable surface matching algorithm to capture the deformation of boundaries of key structures (cortical surface, ventricles and tumor) throughout the neurosurgical procedure, and a linear finite element elastic model to infer a volumetric deformation. The boundary data are extracted from intraoperative MR images using a real-time intraoperative segmentation algorithm. The algorithm has been applied to a sequence of intraoperative MR images of the brain exhibiting brain shift and tumor resection. Our results characterize the brain shift after opening of the dura and at the different stages of tumor resection, and brain swelling afterwards. Analysis of the average deformation capture was assessed by comparing landmarks identified manually and the results indicate an accuracy of 0.7+/-0.6 mm (mean+/-S.D.) for boundary surface landmarks, of 0.9+/-0.6 mm for landmarks inside the boundary surfaces, and 1.6+/-0.9 mm for landmarks in the vicinity of the tumor.     

From colour to tissue histology: Physics-based interpretation of images of pigmented skin lesions

Through an understanding of the image formation process, diagnostically important facts about the internal structure and composition of pigmented skin lesions can be derived from their colour images. A physics-based model of tissue colouration provides a cross-reference between image colours and the underlying histological parameters. It is constructed by computing the spectral composition of light remitted from the skin given parameters specifying its structure and optical properties. The model is representative of all the normal human skin colours, irrespective of racial origin, age or gender. Abnormal skin colours do not conform to this model and thus can be detected. Once the model is constructed, for each pixel in a colour image its histological parameters are computed from the model. Represented as images, these 'parametric maps' show the concentration of dermal and epidermal melanin, blood and collagen thickness across the imaged skin as well as locations where abnormal colouration exists. In a clinical study the parametric maps were used by a clinician to detect the presence of malignant melanoma in a set of 348 pigmented lesions imaged using a commercial device, the SIAscope. Logistic regression identified the presence of melanin in the dermis, the abnormal distribution of blood within the lesion and the lesion size as the most diagnostically informative features. Classification based on these features showed 80.1% sensitivity and 82.7% specificity in melanoma detection.     

Localization of anatomical point landmarks in 3D medical images by fitting 3D parametric intensity models

We introduce a new approach for the localization of 3D anatomical point landmarks. This approach is based on 3D parametric intensity models which are directly fitted to 3D images. To efficiently model tip-like, saddle-like, and sphere-like anatomical structures we introduce analytic intensity models based on the Gaussian error function in conjunction with 3D rigid transformations as well as deformations. To select a suitable size of the region-of-interest (ROI) where model fitting is performed, we also propose a new scheme for automatic selection of an optimal 3D ROI size based on the dominant gradient direction. In addition, to achieve a higher level of automation we present an algorithm for automatic initialization of the model parameters. Our approach has been successfully applied to accurately localize anatomical landmarks in 3D synthetic data as well as 3D MR and 3D CT image data. We have also compared the experimental results with the results of a previously proposed 3D differential approach. It turns out that the new approach significantly improves the localization accuracy.     

Landmark constellation models for medical image content identification and localization

Purpose

Many medical imaging tasks require the detection and localization of anatomical landmarks, for example for the initialization of model-based segmentation or to detect anatomical regions present in an image. A large number of landmark and object localization methods have been described in the literature. The detection of single landmarks may be insufficient to achieve robust localization across a variety of imaging settings and subjects. Furthermore, methods like the generalized Hough transform yield the most likely location of an object, but not an indication whether or not the landmark was actually present in the image.

Methods

For these reasons, we developed a simple and computationally efficient method combining localization results from multiple landmarks to achieve robust localization and to compute a localization confidence measure. For each anatomical region, we train a constellation model indicating the mean relative locations and location variability of a set of landmarks. This model is registered to the landmarks detected in a test image via point-based registration, using closed-form solutions. Three different outlier suppression schemes are compared, two using iterative re-weighting based on the residual landmark registration errors and the third being a variant of RANSAC. The mean weighted residual registration error serves as a confidence measure to distinguish true from false localization results. The method is optimized and evaluated on synthetic data, evaluating both the localization accuracy and the ability to classify good from bad registration results based on the residual registration error.

Results

Two application examples are presented: the identification of the imaged anatomical region in trauma CT scans and the initialization of model-based segmentation for C-arm CT scans with different target regions. The identification of the target region with the presented method was in 96 % of the cases correct.

Conclusion

The presented method is a simple solution for combining multiple landmark localization results. With appropriate parameters, outlier suppression clearly improves the localization performance over model registration without outlier suppression. The optimum choice of method and parameters depends on the expected level of noise and outliers in the application at hand, as well as on the focus on localization, classification, or both. The method allows detecting and localizing anatomical fields of view in medical images and is well suited to support a wide range of applications comprising image content identification, anatomical navigation and visualization, or initializing the pose of organ shape models.

     

A 3D computer graphics approach to brachytherapy planning

Intravascular brachytherapy (IVB) can significantly reduce the risk of restenosis after interventional treatment of stenotic arteries, if planned and applied correctly. In order to facilitate computer-based IVB planning, a three-dimensional reconstruction of the stenotic artery based on intravascular ultrasound (IVUS) sequences is desirable. For this purpose, the frames of the IVUS sequence are properly aligned in space, possible gaps inbetween the IVUS frames are filled by interpolation with radial basis functions known from scattered data interpolation. The alignment procedure uses additional information which is obtained from biplane X-ray angiography performed simultaneously during the capturing of the IVUS sequence. After IVUS images and biplane angiography data are acquired from the patient, the vessel-wall borders and the IVUS catheter are detected by an active contour algorithm. Next, the twist (relative orientation) between adjacent IVUS frames is determined by a sequential triangulation method. The absolute orientation of each frame is established by a stochastic analysis based on anatomical landmarks. Finally, the reconstructed 3D vessel model is visualized by methods of combined volume and polygon rendering. The reconstruction is then used for the computation of the radiation-distribution within the tissue, emitted from a beta-radiation source. All these steps are performed during the percutaneous intervention.     

Automatic estimation of aortic and mitral valve displacements in dynamic CTA with 4D graph-cuts

The location of the mitral and aortic valves in dynamic cardiac imaging is useful for extracting functional derived parameters such as ejection fraction, valve excursions, and global longitudinal strain, and when performing anatomical structures tracking using slice following or valve intervention's planning. Completely automatic segmentation methods are still challenging tasks because of their fast movements and the different positions that prevent good visibility of the leaflets along the full cardiac cycle. In this article, we propose a processing pipeline to track the displacement of the aortic and mitral valve annuli from high-resolution cardiac four-dimensional computed tomographic angiography (4D-CTA). The proposed method is based on the dynamic separation of left ventricle, left atrium and aorta using statistical shape modeling and an energy minimization algorithm based on graph-cuts and has been evaluated on a set of 15 electrocardiography-gated 4D-CTAs. We report a mean agreement distance between manual annotations and our proposed method of 2.52±1.06 mm for the mitral annulus and 2.00±0.69 mm for the aortic valve annulus based on valve locations detected from manual anatomical landmarks. In addition, we show the effect of detecting the valvular planes on derived functional parameters (ejection fraction, global longitudinal strain, and excursions of the mitral and aortic valves).     

Global localization of 3D anatomical structures by pre-filtered Hough Forests and discrete optimization

The accurate localization of anatomical landmarks is a challenging task, often solved by domain specific approaches. We propose a method for the automatic localization of landmarks in complex, repetitive anatomical structures.The key idea is to combine three steps: (1) a classifier for pre-filtering anatomical landmark positions that (2) are refined through a Hough regression model, together with (3) a parts-based model of the global landmark topology to select the final landmark positions. During training landmarks are annotated in a set of example volumes. A classifier learns local landmark appearance, and Hough regressors are trained to aggregate neighborhood information to a precise landmark coordinate position. A non-parametric geometric model encodes the spatial relationships between the landmarks and derives a topology which connects mutually predictive landmarks. During the global search we classify all voxels in the query volume, and perform regression-based agglomeration of landmark probabilities to highly accurate and specific candidate points at potential landmark locations. We encode the candidates’ weights together with the conformity of the connecting edges to the learnt geometric model in a Markov Random Field (MRF). By solving the corresponding discrete optimization problem, the most probable location for each model landmark is found in the query volume.We show that this approach is able to consistently localize the model landmarks despite the complex and repetitive character of the anatomical structures on three challenging data sets (hand radiographs, hand CTs, and whole body CTs), with a median localization error of 0.80 mm, 1.19 mm and 2.71 mm, respectively.     

Lumbar Ultrasound Image Feature Extraction and Classification with Support Vector Machine

Needle entry site localization remains a challenge for procedures that involve lumbar puncture, for example, epidural anesthesia. To solve the problem, we have developed an image classification algorithm that can automatically identify the bone/interspinous region for ultrasound images obtained from lumbar spine of pregnant patients in the transverse plane. The proposed algorithm consists of feature extraction, feature selection and machine learning procedures. A set of features, including matching values, positions and the appearance of black pixels within pre-defined windows along the midline, were extracted from the ultrasound images using template matching and midline detection methods. A support vector machine was then used to classify the bone images and interspinous images. The support vector machine model was trained with 1,040 images from 26 pregnant subjects and tested on 800 images from a separate set of 20 pregnant patients. A success rate of 95.0% on training set and 93.2% on test set was achieved with the proposed method. The trained support vector machine model was further tested on 46 off-line collected videos, and successfully identified the proper needle insertion site (interspinous region) in 45 of the cases. Therefore, the proposed method is able to process the ultrasound images of lumbar spine in an automatic manner, so as to facilitate the anesthetists' work of identifying the needle entry site.     

Personalization of a cardiac electromechanical model using reduced order unscented Kalman filtering from regional volumes

Patient-specific cardiac modeling can help in understanding pathophysiology and therapy planning. However it requires to combine functional and anatomical data in order to build accurate models and to personalize the model geometry, kinematics, electrophysiology and mechanics. Personalizing the electromechanical coupling from medical images is a challenging task. We use the Bestel–Clément–Sorine (BCS) electromechanical model of the heart, which provides reasonable accuracy with a reasonable number of parameters (14 for each ventricle) compared to the available clinical data at the organ level. We propose a personalization strategy from cine MRI data in two steps. We first estimate global parameters with an automatic calibration algorithm based on the Unscented Transform which allows to initialize the parameters while matching the volume and pressure curves. In a second step we locally personalize the contractilities of all AHA (American Heart Association) zones of the left ventricle using the reduced order unscented Kalman filtering on Regional Volumes. This personalization strategy was validated synthetically and tested successfully on eight healthy and three pathological cases.     

Automatic detection of landmarks for the analysis of a reduction of supracondylar fractures of the humerus

An accurate identification of bone features is required by modern orthopedics to improve patient recovery. The analysis of landmarks enables the planning of a fracture reduction surgery, designing prostheses or fixation devices, and showing deformities accurately. The recognition of these features was previously performed manually. However, this long and tedious process provided insufficient accuracy. In this paper, we propose a geometrically-based algorithm that automatically detects the most significant landmarks of a humerus. By employing contralateral images of the upper limb, a side-to-side study of the landmarks is also conducted to analyze the goodness of supracondylar fracture reductions. We conclude that a reduction can be classified by only considering the detected landmarks. In addition, our technique does not require a prior training, thus becoming a reliable alternative to treat this kind of fractures.     

Optimisation of the 3D MDEFT sequence for anatomical brain imaging: technical implications at 1.5 and 3 T

An algorithm for the optimisation of 3D Modified Driven Equilibrium Fourier Transform (MDEFT) sequences for T1-weighted anatomical brain imaging is presented. Imaging parameters are optimised for a clinical whole body scanner and a clinical head scanner operating at 1.5 and 3 T, respectively. In vivo studies show that the resulting sequences allow for the whole brain acquisition of anatomical scans with an isotropic resolution of 1 mm and high contrast-to-noise ratio (CNR) in an acceptable scan time of 12 min. Typical problems related to the scanner-specific hardware configurations are discussed in detail, especially the occurrence of flow artefacts in images acquired with head transmit coils and the enhancement of scalp intensities in images acquired with phased array receive coils. It is shown both theoretically and experimentally that these problems can be avoided by using spin tagging and fat saturation.     

Landmark constrained genus-one surface Teichmüller map applied to surface registration in medical imaging

We address the registration problem of genus-one surfaces (such as vertebrae bones) with prescribed landmark constraints. The high-genus topology of the surfaces makes it challenging to obtain a unique and bijective surface mapping that matches landmarks consistently. This work proposes to tackle this registration problem using a special class of quasi-conformal maps called Teichmüller maps (T-Maps). A landmark constrained T-Map is the unique mapping between genus-1 surfaces that minimizes the maximal conformality distortion while matching the prescribed feature landmarks. Existence and uniqueness of the landmark constrained T-Map are theoretically guaranteed. This work presents an iterative algorithm to compute the T-Map. The main idea is to represent the set of diffeomorphism using the Beltrami coefficients (BC). The BC is iteratively adjusted to an optimal one, which corresponds to our desired T-Map that matches the prescribed landmarks and satisfies the periodic boundary condition on the universal covering space. Numerical experiments demonstrate the effectiveness of our proposed algorithm. The method has also been applied to register vertebrae bones with prescribed landmark points and curves, which gives accurate surface registrations.