Metric hashing forests

In this paper, we propose metric Hashing Forests (mHF) which is a supervised variant of random forests tailored for the task of nearest neighbor retrieval through hashing. This is achieved by training independent hashing trees that parse and encode the feature space such that local class neighborhoods are preserved and encoded with similar compact binary codes. At the level of each internal node, locality preserving projections are employed to project data to a latent subspace, where separability between dissimilar points is enhanced. Following which, we define an oblique split that maximally preserves this separability and facilitates defining local neighborhoods of similar points. By incorporating the inverse-lookup search scheme within the mHF, we can then effectively mitigate pairwise neuron similarity comparisons, which allows for scalability to massive databases with little additional time overhead. Exhaustive experimental validations on 22,265 neurons curated from over 120 different archives demonstrate the superior efficacy of mHF in terms of its retrieval performance and precision of classification in contrast to state-of-the-art hashing and metric learning based methods. We conclude that the proposed method can be utilized effectively for similarity-preserving retrieval and categorization in large neuron databases.     

Groupwise surface correspondence by optimization: representation and regularization

Groupwise optimization of correspondence across a set of unlabelled examples of shapes or images is a well-established technique that has been shown to produce quantitatively better models than other approaches. However, the computational cost of the optimization is high, leading to long convergence times. In this paper, we show how topologically non-trivial shapes can be mapped to regular grids, hence represented in terms of vector-valued functions defined on these grids (the shape image representation). This leads to an initial reduction in computational complexity. We also consider the question of regularization, and show that by borrowing ideas from image registration, it is possible to build a non-parametric, fluid regularizer for shapes, without losing the computational gain made by the use of shape images. We show that this non-parametric regularization leads to a further considerable gain, when compared to parametric regularization methods. Quantitative evaluation is performed on biological datasets, and shown to yield a substantial decrease in convergence time, with no loss of model quality.     

Deep metric learning-based image retrieval system for chest radiograph and its clinical applications in COVID-19

In recent years, deep learning-based image analysis methods have been widely applied in computer-aided detection, diagnosis and prognosis, and has shown its value during the public health crisis of the novel coronavirus disease 2019 (COVID-19) pandemic. Chest radiograph (CXR) has been playing a crucial role in COVID-19 patient triaging, diagnosing and monitoring, particularly in the United States. Considering the mixed and unspecific signals in CXR, an image retrieval model of CXR that provides both similar images and associated clinical information can be more clinically meaningful than a direct image diagnostic model. In this work we develop a novel CXR image retrieval model based on deep metric learning. Unlike traditional diagnostic models which aim at learning the direct mapping from images to labels, the proposed model aims at learning the optimized embedding space of images, where images with the same labels and similar contents are pulled together. The proposed model utilizes multi-similarity loss with hard-mining sampling strategy and attention mechanism to learn the optimized embedding space, and provides similar images, the visualizations of disease-related attention maps and useful clinical information to assist clinical decisions. The model is trained and validated on an international multi-site COVID-19 dataset collected from 3 different sources. Experimental results of COVID-19 image retrieval and diagnosis tasks show that the proposed model can serve as a robust solution for CXR analysis and patient management for COVID-19. The model is also tested on its transferability on a different clinical decision support task for COVID-19, where the pre-trained model is applied to extract image features from a new dataset without any further training. The extracted features are then combined with COVID-19 patient's vitals, lab tests and medical histories to predict the possibility of airway intubation in 72 hours, which is strongly associated with patient prognosis, and is crucial for patient care and hospital resource planning. These results demonstrate our deep metric learning based image retrieval model is highly efficient in the CXR retrieval, diagnosis and prognosis, and thus has great clinical value for the treatment and management of COVID-19 patients.     

Supervised graph hashing for histopathology image retrieval and classification

In pathology image analysis, morphological characteristics of cells are critical to grade many diseases. With the development of cell detection and segmentation techniques, it is possible to extract cell-level information for further analysis in pathology images. However, it is challenging to conduct efficient analysis of cell-level information on a large-scale image dataset because each image usually contains hundreds or thousands of cells. In this paper, we propose a novel image retrieval based framework for large-scale pathology image analysis. For each image, we encode each cell into binary codes to generate image representation using a novel graph based hashing model and then conduct image retrieval by applying a group-to-group matching method to similarity measurement. In order to improve both computational efficiency and memory requirement, we further introduce matrix factorization into the hashing model for scalable image retrieval. The proposed framework is extensively validated with thousands of lung cancer images, and it achieves 97.98% classification accuracy and 97.50% retrieval precision with all cells of each query image used.     

Case-based similar image retrieval for weakly annotated large histopathological images of malignant lymphoma using deep metric learning

In the present study, we propose a novel case-based similar image retrieval (SIR) method for hematoxylin and eosin (H&E) stained histopathological images of malignant lymphoma. When a whole slide image (WSI) is used as an input query, it is desirable to be able to retrieve similar cases by focusing on image patches in pathologically important regions such as tumor cells. To address this problem, we employ attention-based multiple instance learning, which enables us to focus on tumor-specific regions when the similarity between cases is computed. Moreover, we employ contrastive distance metric learning to incorporate immunohistochemical (IHC) staining patterns as useful supervised information for defining appropriate similarity between heterogeneous malignant lymphoma cases. In the experiment with 249 malignant lymphoma patients, we confirmed that the proposed method exhibited higher evaluation measures than the baseline case-based SIR methods. Furthermore, the subjective evaluation by pathologists revealed that our similarity measure using IHC staining patterns is appropriate for representing the similarity of H&E stained tissue images for malignant lymphoma.     

Joint optimization of segmentation and shape prior from level-set-based statistical shape model,and its application to the automated segmentation of abdominal organs

The goal of this study is to provide a theoretical framework for accurately optimizing the segmentation energy considering all of the possible shapes generated from the level-set-based statistical shape model (SSM). The proposed algorithm solves the well-known open problem, in which a shape prior may not be optimal in terms of an objective functional that needs to be minimized during segmentation. The algorithm allows the selection of an optimal shape prior from among all possible shapes generated from an SSM by conducting a branch-and-bound search over an eigenshape space. The proposed algorithm does not require predefined shape templates or the construction of a hierarchical clustering tree before graph-cut segmentation. It jointly optimizes an objective functional in terms of both the shape prior and segmentation labeling, and finds an optimal solution by considering all possible shapes generated from an SSM. We apply the proposed algorithm to both pancreas and spleen segmentation using multiphase computed tomography volumes, and we compare the results obtained with those produced by a conventional algorithm employing a branch-and-bound search over a search tree of predefined shapes, which were sampled discretely from an SSM. The proposed algorithm significantly improves the segmentation performance in terms of the Jaccard index and Dice similarity index. In addition, we compare the results with the state-of-the-art multiple abdominal organs segmentation algorithm, and confirmed that the performances of both algorithms are comparable to each other. We discuss the high computational efficiency of the proposed algorithm, which was determined experimentally using a normalized number of traversed nodes in a search tree, and the extensibility of the proposed algorithm to other SSMs or energy functionals.     

A regular grid-based Hough transform for the extraction of urban features using high-resolution satellite images

The Hough transform (HT) is a widely used line extraction method to detect the boundary of urban features. However, the HT has some problems, such as high computational costs and omissions of lines in remote sensing images. In this study, a robust and improved Hough line extraction method, which uses a regular grid and adjacent information from the base grid cell and its neighbouring grid cells, is proposed. The proposed method efficiently delineates the lines of urban features, without any line omissions. The regular grid aids the direct determination of the location and size of the transform region and decreases the computational cost. In addition, the adjacent information is useful for line connections and removes the need for complicated geometric factors. The proposed regular grid-based Hough transform (RGHT) was compared with the standard Hough transform (SHT). The results showed that the proposed method extracted evenly distributed lines over the entire image at a low computational cost. Furthermore, the proposed method extracted rectangular and curved lines of buildings and roads sites better than the SHT method, without omitting portions of the urban features.     

Locally Linear Diffeomorphic Metric Embedding (LLDME) for surface-based anatomical shape modeling

This paper presents the algorithm, Locally Linear Diffeomorphic Metric Embedding (LLDME), for constructing efficient and compact representations of surface-based brain shapes whose variations are characterized using Large Deformation Diffeomorphic Metric Mapping (LDDMM). Our hypothesis is that the shape variations in the infinite-dimensional diffeomorphic metric space can be captured by a low-dimensional space. To do so, traditional Locally Linear Embedding (LLE) that reconstructs a data point from its neighbors in Euclidean space is extended to LLDME that requires interpolating a shape from its neighbors in the infinite-dimensional diffeomorphic metric space. This is made possible through the conservation law of momentum derived from LDDMM. It indicates that initial momentum, a linear transformation of the initial velocity of diffeomorphic flows, at a fixed template shape determines the geodesic connecting the template to a subject's shape in the diffeomorphic metric space and becomes the shape signature of an individual subject. This leads to the compact linear representation of the nonlinear diffeomorphisms in terms of the initial momentum. Since the initial momentum is in a linear space, a shape can be approximated by a linear combination of its neighbors in the diffeomorphic metric space. In addition, we provide efficient computations for the metric distance between two shapes through the first order approximation of the geodesic using the initial momentum as well as for the reconstruction of a shape given its low-dimensional Euclidean coordinates using the geodesic shooting with the initial momentum as the initial condition. Experiments are performed on the hippocampal shapes of 302 normal subjects across the whole life span (18-94years). Compared with Principal Component Analysis and ISOMAP, LLDME provides the most compact and efficient representation of the age-related hippocampal shapes. Even though the hippocampal volumes among young adults are as variable as those in older adults, LLDME disentangles the hippocampal local shape variation from the hippocampal size and thus reveals the nonlinear relationship of the hippocampal morphometry with age.     

Computational tissue staining of non-linear multimodal imaging using supervised and unsupervised deep learning

Hematoxylin and Eosin (H&E) staining is the ’gold-standard’ method in histopathology. However, standard H&E staining of high-quality tissue sections requires long sample preparation times including sample embedding, which restricts its application for ’real-time’ disease diagnosis. Due to this reason, a label-free alternative technique like non-linear multimodal (NLM) imaging, which is the combination of three non-linear optical modalities including coherent anti-Stokes Raman scattering, two-photon excitation fluorescence and second-harmonic generation, is proposed in this work. To correlate the information of the NLM images with H&E images, this work proposes computational staining of NLM images using deep learning models in a supervised and an unsupervised approach. In the supervised and the unsupervised approach, conditional generative adversarial networks (CGANs) and cycle conditional generative adversarial networks (cycle CGANs) are used, respectively. Both CGAN and cycle CGAN models generate pseudo H&E images, which are quantitatively analyzed based on mean squared error, structure similarity index and color shading similarity index. The mean of the three metrics calculated for the computationally generated H&E images indicate significant performance. Thus, utilizing CGAN and cycle CGAN models for computational staining is beneficial for diagnostic applications without performing a laboratory-based staining procedure. To the author’s best knowledge, it is the first time that NLM images are computationally stained to H&E images using GANs in an unsupervised manner.     

自适应超分辨率图像的重建算法（英文）

背景：超分辨率重建已经在视频、遥感等许多领域内的到广泛的研究与应用。目的：介绍一种自适应超分辨率重建算法,以期从序列低分辨率图像中重建出高分辨率图像。方法：采用常数λ=2/3作为正则化参数和自适应步长作为第一种方案。第二种方案充分考虑到低分辨率图像中的运动误差估计、点扩散函数以及加性高斯白噪声对重建算法的影响。实验构造出新的非线性自适应正则化函数,进而利用实验方法分析代价函数的凸性。通过数学理论,根据代价函数凸性实验得到自适应步长因子,从而改进了图像的空间分辨率和算法的收敛速度。结果与结论：为验证此算法的有效性,采用光学图像进行实验。方案二图像峰值信噪比增高,其收敛速度为方案一的2倍以上;方案二的平均计算需要的时间为68.25s。结果证实,自适应超分辨率图像重建算法对图像分辨率和迭代的收敛速度均改善显著,其稳定性较好。     

Wavelet optimization for content-based image retrieval in medical databases

We propose in this article a content-based image retrieval (CBIR) method for diagnosis aid in medical fields. In the proposed system, images are indexed in a generic fashion, without extracting domain-specific features: a signature is built for each image from its wavelet transform. These image signatures characterize the distribution of wavelet coefficients in each subband of the decomposition. A distance measure is then defined to compare two image signatures and thus retrieve the most similar images in a database when a query image is submitted by a physician. To retrieve relevant images from a medical database, the signatures and the distance measure must be related to the medical interpretation of images. As a consequence, we introduce several degrees of freedom in the system so that it can be tuned to any pathology and image modality. In particular, we propose to adapt the wavelet basis, within the lifting scheme framework, and to use a custom decomposition scheme. Weights are also introduced between subbands. All these parameters are tuned by an optimization procedure, using the medical grading of each image in the database to define a performance measure. The system is assessed on two medical image databases: one for diabetic retinopathy follow up and one for screening mammography, as well as a general purpose database. Results are promising: a mean precision of 56.50%, 70.91% and 96.10% is achieved for these three databases, when five images are returned by the system.     

From macro-scale to micro-scale computational anatomy: a perspective on the next 20 years

This paper gives our perspective on the next two decades of computational anatomy, which has made great strides in the recognition and understanding of human anatomy from conventional clinical images. The results from this field are now used in a variety of medical applications, including quantitative analysis of organ shapes, interventional assistance, surgical navigation, and population analysis. Several anatomical models have also been used in computational anatomy, and these mainly target millimeter-scale shapes. For example, liver-shape models are almost completely modeled at the millimeter scale, and shape variations are described at such scales. Most clinical 3D scanning devices have had just under 1 or 0.5 mm per voxel resolution for over 25 years, and this resolution has not changed drastically in that time. Although Z-axis (head-to-tail direction) resolution has been drastically improved by the introduction of multi-detector CT scanning devices, in-plane resolutions have not changed very much either. When we look at human anatomy, we can see different anatomical structures at different scales. For example, pulmonary blood vessels and lung lobes can be observed in millimeter-scale images. If we take 10-µm-scale images of a lung specimen, the alveoli and bronchiole regions can be located in them. Most work in millimeter-scale computational anatomy has been done by the medical-image analysis community. In the next two decades, we encourage our community to focus on micro-scale computational anatomy. In this perspective paper, we briefly review the achievements of computational anatomy and its impacts on clinical applications; furthermore, we show several possibilities from the viewpoint of microscopic computational anatomy by discussing experimental results from our recent research activities.     

Bayesian logistic shape model inference: Application to cochlear image segmentation

Incorporating shape information is essential for the delineation of many organs and anatomical structures in medical images. While previous work has mainly focused on parametric spatial transformations applied to reference template shapes, in this paper, we address the Bayesian inference of parametric shape models for segmenting medical images with the objective of providing interpretable results. The proposed framework defines a likelihood appearance probability and a prior label probability based on a generic shape function through a logistic function. A reference length parameter defined in the sigmoid controls the trade-off between shape and appearance information. The inference of shape parameters is performed within an Expectation-Maximisation approach in which a Gauss-Newton optimization stage provides an approximation of the posterior probability of the shape parameters. This framework is applied to the segmentation of cochlear structures from clinical CT images constrained by a 10-parameter shape model. It is evaluated on three different datasets, one of which includes more than 200 patient images. The results show performances comparable to supervised methods and better than previously proposed unsupervised ones. It also enables an analysis of parameter distributions and the quantification of segmentation uncertainty, including the effect of the shape model.     

Automatic Image Guidance for Assessment of Placenta Location in Ultrasound Video Sweeps

Ultrasound-based assistive tools are aimed at reducing the high skill needed to interpret a scan by providing automatic image guidance. This may encourage uptake of ultrasound (US) clinical assessments in rural settings in low- and middle-income countries (LMICs), where well-trained sonographers can be scarce. This paper describes a new method that automatically generates an assistive video overlay to provide image guidance to a user to assess placenta location. The user captures US video by following a sweep protocol that scans a U-shape on the lower maternal abdomen. The sweep trajectory is simple and easy to learn. We initially explore a 2-D embedding of placenta shapes, mapping manually segmented placentas in US video frames to a 2-D space. We map 2013 frames from 11 videos. This provides insight into the spectrum of placenta shapes that appear when using the sweep protocol. We propose classification of the placenta shapes from three observed clusters: complex, tip and rectangular. We use this insight to design an effective automatic segmentation algorithm, combining a U-Net with a CRF-RNN module to enhance segmentation performance with respect to placenta shape. The U-Net + CRF-RNN algorithm automatically segments the placenta and maternal bladder. We assess segmentation performance using both area and shape metrics. We report results comparable to the state-of-the-art for automatic placenta segmentation on the Dice metric, achieving 0.83 ± 0.15 evaluated on 2127 frames from 10 videos. We also qualitatively evaluate 78,308 frames from 135 videos, assessing if the anatomical outline is correctly segmented. We found that addition of the CRF-RNN improves over a baseline U-Net when faced with a complex placenta shape, which we observe in our 2-D embedding, up to 14% with respect to the percentage shape error. From the segmentations, an assistive video overlay is automatically constructed that (i) highlights the placenta and bladder, (ii) determines the lower placenta edge and highlights this location as a point and (iii) labels a 2-cm clearance on the lower placenta edge. The 2-cm clearance is chosen to satisfy current clinical guidelines. We propose to assess the placenta location by comparing the 2-cm region and the bottom of the bladder, which represents a coarse localization of the cervix. Anatomically, the bladder must sit above the cervix region. We present proof-of-concept results for the video overlay.     

Reducing annotation effort in digital pathology: A Co-Representation learning framework for classification tasks

Classification of digital pathology images is imperative in cancer diagnosis and prognosis. Recent advancements in deep learning and computer vision have greatly benefited the pathology workflow by developing automated solutions for classification tasks. However, the cost and time for acquiring high quality task-specific large annotated training data are subject to intra- and inter-observer variability, thus challenging the adoption of such tools. To address these challenges, we propose a classification framework via co-representation learning to maximize the learning capability of deep neural networks while using a reduced amount of training data. The framework captures the class-label information and the local spatial distribution information by jointly optimizing a categorical cross-entropy objective and a deep metric learning objective respectively. A deep metric learning objective is incorporated to enhance the classification, especially in the low training data regime. Further, a neighborhood-aware multiple similarity sampling strategy, and a soft-multi-pair objective that optimizes interactions between multiple informative sample pairs, is proposed to accelerate deep metric learning. We evaluate the proposed framework on five benchmark datasets from three digital pathology tasks, i.e., nuclei classification, mitosis detection, and tissue type classification. For all the datasets, our framework achieves state-of-the-art performance when using approximately only 50% of the training data. On using complete training data, the proposed framework outperforms the state-of-the-art on all the five datasets.     

Replacing pixel representations by point-function schemes for reducing discretization error in ill-posed remote sensing problems,with examples from cloud tomography

Because of their simplicity and low computational cost, discretizations based on pixels have held sway in remote sensing since its inception. Yet functional representations are clearly superior in many applications, for example when combining retrievals from dissimilar remote sensing instruments. Using cloud tomography as an example, this letter shows that a point-function discretization scheme based on linear interpolation can reduce retrieval error of cloud water content up to 40% compared to a conventional pixel scheme. This improvement is particularly marked because cloud tomography, like the vast majority of remote sensing problems, is ill-posed and thus a small inaccuracy in the formulation of the retrieval problem, such as discretization error, can cause a large error in the retrievals.     

Designing user interfaces to enhance human interpretation of medical content-based image retrieval: application to PET-CT images

Purpose Content-based image retrieval (CBIR) in medicine has been demonstrated to improve evidence-based diagnosis, education, and teaching. However, the low clinical adoption of CBIR is partially because the focus of most studies has been the development of feature extraction and similarity measurement algorithms with limited work on facilitating better understanding of the similarity between complex volumetric and multi-modality medical images. In this paper, we present a method for defining user interfaces (UIs) that enable effective human user interpretation of retrieved images. Methods We derived a set of visualisation and interaction requirements based on the characteristics of modern volumetric medical images. We implemented a UI that visualised multiple views of a single image, displayed abstractions of image data, and provided access to supplementary non-image data. We also defined interactions for refining the search and visually indicating the similarities between images. We applied the UI for the retrieval of multi-modality positron emission tomography and computed tomography (PET-CT) images. We conducted a user survey to evaluate the capabilities of our UI. Results Our proposed method obtained a high rating ( $\ge $ 4 out of 5) in the majority of survey questions. In particular, the survey responses indicated the UI presented all the information necessary to understand the retrieved images, and did so in an intuitive manner. Conclusion Our proposed UI design improved the ability of users to interpret and understand the similarity between retrieved PET-CT images. The implementation of CBIR UIs designed to assist human interpretation could facilitate wider adoption of medical CBIR systems.     

A spline-based non-linear diffeomorphism for multimodal prostate registration

This paper presents a novel method for non-rigid registration of transrectal ultrasound and magnetic resonance prostate images based on a non-linear regularized framework of point correspondences obtained from a statistical measure of shape-contexts. The segmented prostate shapes are represented by shape-contexts and the Bhattacharyya distance between the shape representations is used to find the point correspondences between the 2D fixed and moving images. The registration method involves parametric estimation of the non-linear diffeomorphism between the multimodal images and has its basis in solving a set of non-linear equations of thin-plate splines. The solution is obtained as the least-squares solution of an over-determined system of non-linear equations constructed by integrating a set of non-linear functions over the fixed and moving images. However, this may not result in clinically acceptable transformations of the anatomical targets. Therefore, the regularized bending energy of the thin-plate splines along with the localization error of established correspondences should be included in the system of equations. The registration accuracies of the proposed method are evaluated in 20 pairs of prostate mid-gland ultrasound and magnetic resonance images. The results obtained in terms of Dice similarity coefficient show an average of 0.980±0.004, average 95% Hausdorff distance of 1.63±0.48mm and mean target registration and target localization errors of 1.60±1.17mm and 0.15±0.12mm respectively.     

Interactive segmentation of abdominal aortic aneurysms in CTA images

A model-based approach to interactive segmentation of abdominal aortic aneurysms from CTA data is presented. After manual delineation of the aneurysm sac in the first slice, the method automatically detects the contour in subsequent slices, using the result from the previous slice as a reference. If an obtained contour is not sufficiently accurate, the user can intervene and provide an additional manual reference contour. The method is inspired by the active shape model (ASM) segmentation scheme (), in which a statistical shape model, derived from corresponding landmark points in manually labeled training images, is fitted to the image in an iterative manner. In our method, a shape model of the contours in two adjacent image slices is progressively fitted to the entire volume. The contour obtained in one slice thus constrains the possible shapes in the next slice. The optimal fit is determined on the basis of multi-resolution gray level models constructed from gray value patches sampled around each landmark. We propose to use the similarity of adjacent image slices for this gray level model, and compare these to single-slice features that are more generally used with ASM. The performance of various image features is evaluated in leave-one-out experiments on 23 data sets. Features that use the similarity of adjacent image slices outperform measures based on single-slice features in all cases. The average number of slices in our datasets is 51, while on average eight manual initializations are required, which decreases operator segmentation time by a factor of 6.