Three-dimensional-generator U-net for dual-resonant scanning multiphoton microscopy image inpainting and denoising

A dual-resonant scanning multiphoton (DRSM) microscope incorporating a tunable acoustic gradient index of refraction lens and a resonant mirror is developed for rapid volumetric bioimaging. It is shown that the microscope achieves a volumetric imaging rate up to 31.25 volumes per second (vps) for a scanning volume of up to 200 × 200 × 100 µm³ with 256 × 256 × 128 voxels. However, the volumetric images have a severe negative signal-to-noise ratio (SNR) as a result of a large number of missing voxels for a large scanning volume and the presence of Lissajous patterning residuals. Thus, a modified three-dimensional (3D)-generator U-Net model trained using simulated microbead images is proposed and used to inpaint and denoise the images. The performance of the 3D U-Net model for bioimaging applications is enhanced by training the model with high-SNR in-vitro drosophila brain images captured using a conventional point scanning multiphoton microscope. The trained model shows the ability to produce clear in-vitro drosophila brain images at a rate of 31.25 vps with a SNR improvement of approximately 20 dB over the original images obtained by the DRSM microscope. The training convergence time of the modified U-Net model is just half that of a general 3D U-Net model. The model thus has significant potential for 3D in-vivo bioimaging transfer learning. Through the assistance of transfer learning, the model can be extended to the restoration of in-vivo drosophila brain images with a high image quality and a rapid training time.     

轮廓波及曲波和小波变换用于显微图像消噪的比较

背景:小波变换只能反映信号的零维奇异性,无法最优表示图像中的线奇异;而且小波变换只存在3个方向,这些都显著影响了它在图像处理领域的应用效果.针对小波变换的缺点,多尺度几何分析理论正在逐步发展,轮廓波变换和曲波变换就是其中的典型代表.目的:定性、定量地比较轮廓波、曲波和小波变换在图像消噪处理中的效果.方法:在简要介绍3种变换基本原理的基础上,比较它们在图像消噪领域的应用,以均方误差和峰值信噪比作为定量指标评价消噪效果,并将其应用于显微镜图像的消噪处理.结果与结论:综合定量评价指标和人眼视觉感受,曲波变换的消噪结果最佳,轮廓波变换效果次之,小波变换效果则不够理想.     

轮廓波及曲波和小波变换用于显微图像消噪的比较

背景：小波变换只能反映信号的零维奇异性,无法最优表示图像中的线奇异;而且小波变换只存在3个方向,这些都显著影响了它在图像处理领域的应用效果。针对小波变换的缺点,多尺度几何分析理论正在逐步发展,轮廓波变换和曲波变换就是其中的典型代表。目的：定性、定量地比较轮廓波、曲波和小波变换在图像消噪处理中的效果。方法：在简要介绍3种变换基本原理的基础上,比较它们在图像消噪领域的应用,以均方误差和峰值信噪比作为定量指标评价消噪效果,并将其应用于显微镜图像的消噪处理。结果与结论：综合定量评价指标和人眼视觉感受,曲波变换的消噪结果最佳,轮廓波变换效果次之,小波变换效果则不够理想。     

Deep learning-based autofocus method enhances image quality in light-sheet fluorescence microscopy

Light-sheet fluorescence microscopy (LSFM) is a minimally invasive and high throughput imaging technique ideal for capturing large volumes of tissue with sub-cellular resolution. A fundamental requirement for LSFM is a seamless overlap of the light-sheet that excites a selective plane in the specimen, with the focal plane of the objective lens. However, spatial heterogeneity in the refractive index of the specimen often results in violation of this requirement when imaging deep in the tissue. To address this issue, autofocus methods are commonly used to refocus the focal plane of the objective-lens on the light-sheet. Yet, autofocus techniques are slow since they require capturing a stack of images and tend to fail in the presence of spherical aberrations that dominate volume imaging. To address these issues, we present a deep learning-based autofocus framework that can estimate the position of the objective-lens focal plane relative to the light-sheet, based on two defocused images. This approach outperforms or provides comparable results with the best traditional autofocus method on small and large image patches respectively. When the trained network is integrated with a custom-built LSFM, a certainty measure is used to further refine the network’s prediction. The network performance is demonstrated in real-time on cleared genetically labeled mouse forebrain and pig cochleae samples. Our study provides a framework that could improve light-sheet microscopy and its application toward imaging large 3D specimens with high spatial resolution.     

MR图像去噪算法研究进展

MRI对医学诊断有重要价值,而去噪是MR图像分析与诊断的基础问题之一.因此,探讨能消除MR图像噪声的方法具有重要的临床意义和应用价值.作者从空间域、变换域及多尺度分析等方面对MR图像去噪算法进行综述,分析和比较其性能,并对医学图像去噪领域的前景和趋势进行展望.     

Improved localization accuracy in stochastic super-resolution fluorescence microscopy by K-factor image deshadowing

Localization of a single fluorescent particle with sub-diffraction-limit accuracy is a key merit in localization microscopy. Existing methods such as photoactivated localization microscopy (PALM) and stochastic optical reconstruction microscopy (STORM) achieve localization accuracies of single emitters that can reach an order of magnitude lower than the conventional resolving capabilities of optical microscopy. However, these techniques require a sparse distribution of simultaneously activated fluorophores in the field of view, resulting in larger time needed for the construction of the full image. In this paper we present the use of a nonlinear image decomposition algorithm termed K-factor, which reduces an image into a nonlinear set of contrast-ordered decompositions whose joint product reassembles the original image. The K-factor technique, when implemented on raw data prior to localization, can improve the localization accuracy of standard existing methods, and also enable the localization of overlapping particles, allowing the use of increased fluorophore activation density, and thereby increased data collection speed. Numerical simulations of fluorescence data with random probe positions, and especially at high densities of activated fluorophores, demonstrate an improvement of up to 85% in the localization precision compared to single fitting techniques. Implementing the proposed concept on experimental data of cellular structures yielded a 37% improvement in resolution for the same super-resolution image acquisition time, and a decrease of 42% in the collection time of super-resolution data with the same resolution.OCIS codes: (180.2520) Fluorescence microscopy, (100.0100) Image processing, (100.3010) Image reconstruction techniques, (100.6640) Superresolution     

Deriving high contrast fluorescence microscopy images through low contrast noisy image stacks

Contrast in fluorescence microscopy images allows for the differentiation between different structures by their difference in intensities. However, factors such as point-spread function and noise may reduce it, affecting its interpretability. We identified that fluctuation of emitters in a stack of images can be exploited to achieve increased contrast when compared to the average and Richardson-Lucy deconvolution. We tested our methods on four increasingly challenging samples including tissue, in which case results were comparable to the ones obtained by structured illumination microscopy in terms of contrast.     

DETECTOR: structural information guided artifact detection for super-resolution fluorescence microscopy image

Super-resolution fluorescence microscopy, with a spatial resolution beyond the diffraction limit of light, has become an indispensable tool to observe subcellular structures at a nanoscale level. To verify that the super-resolution images reflect the underlying structures of samples, the development of robust and reliable artifact detection methods has received widespread attention. However, the existing artifact detection methods are prone to report false alert artifacts because it relies on absolute intensity mismatch between the wide-field image and resolution rescaled super-resolution image. To solve this problem, we proposed DETECTOR, a structural information-guided artifact detection method for super-resolution images. It detects artifacts by computing the structural dissimilarity between the wide-field image and the resolution rescaled super-resolution image. To focus on structural similarity, we introduce a weight mask to weaken the influence of strong autofluorescence background and proposed a structural similarity index for super-resolution images, named MASK-SSIM. Simulations and experimental results demonstrated that compared with the state-of-the-art methods, DETECTOR has advantages in detecting structural artifacts in super-resolution images. It is especially suitable for wide-field images with strong autofluorescence background and super-resolution images of single molecule localization microscopy (SMLM). DETECTOR has extreme sensitivity to the weak signal region. Moreover, DETECTOR can guide data collection and parameter tuning during image reconstruction.     

Non-rigid multi-frame registration of cell nuclei in live cell fluorescence microscopy image data

The analysis of the motion of subcellular particles in live cell microscopy images is essential for understanding biological processes within cells. For accurate quantification of the particle motion, compensation of the motion and deformation of the cell nucleus is required. We introduce a non-rigid multi-frame registration approach for live cell fluorescence microscopy image data. Compared to existing approaches using pairwise registration, our approach exploits information from multiple consecutive images simultaneously to improve the registration accuracy. We present three intensity-based variants of the multi-frame registration approach and we investigate two different temporal weighting schemes. The approach has been successfully applied to synthetic and live cell microscopy image sequences, and an experimental comparison with non-rigid pairwise registration has been carried out.     

Illumination angle correction during image acquisition in light-sheet fluorescence microscopy using deep learning

Light-sheet fluorescence microscopy (LSFM) is a high-speed imaging technique that provides optical sectioning with reduced photodamage. LSFM is routinely used in life sciences for live cell imaging and for capturing large volumes of cleared tissues. LSFM has a unique configuration, in which the illumination and detection paths are separated and perpendicular to each other. As such, the image quality, especially at high resolution, largely depends on the degree of overlap between the detection focal plane and the illuminating beam. However, spatial heterogeneity within the sample, curved specimen boundaries, and mismatch of refractive index between tissues and immersion media can refract the well-aligned illumination beam. This refraction can cause extensive blur and non-uniform image quality over the imaged field-of-view. To address these issues, we tested a deep learning-based approach to estimate the angular error of the illumination beam relative to the detection focal plane. The illumination beam was then corrected using a pair of galvo scanners, and the correction significantly improved the image quality across the entire field-of-view. The angular estimation was based on calculating the defocus level on a pixel level within the image using two defocused images. Overall, our study provides a framework that can correct the angle of the light-sheet and improve the overall image quality in high-resolution LSFM 3D image acquisition.     

Parametric comparison between sparsity-based and deep learning-based image reconstruction of super-resolution fluorescence microscopy

Sparsity-based and deep learning-based image reconstruction algorithms are two promising approaches to accelerate the image acquisition process for localization-based super-resolution microscopy, by allowing a higher density of fluorescing emitters to be imaged in a single frame. Despite the surging popularity, a comprehensive parametric study guiding the practical applications of sparsity-based and deep learning-based image reconstruction algorithms is yet to be conducted. In this study, we examined the performance of sparsity- and deep learning-based algorithms in reconstructing super-resolution images using simulated fluorescent microscopy images. The simulated images were synthesized with varying levels of sparsity and connectivity. We found the deep learning-based VDSR recovers image faster, with a higher recall rate and localization accuracy. The sparsity-based SPIDER recovers more zero pixels truthfully. We also compared the two algorithms using images acquired from a real super-resolution experiment, yielding results agreeing with the results from the evaluation using simulated images. We concluded that VDSR is preferable when accurate emitter localization is needed while SPIDER is more suitable when evaluation of the number of emitters is critical.     

Hyperspectral image denoising via low-rank matrix recovery

Based on low-rank matrix recovery theory, we propose a novel method to remove the hyperspectral image noise. To robustly handle the outliers in hyperspectral images, we first build a hybrid noise model for the hyperspectral images. Then, the noise removal is achieved via two stages. In the first stage, the main fine-image features are first separated from the noise via principal component analysis (PCA) due to its good performance in signal/noise decorrelation. In the second stage, the noise removal is conducted in the low-energy PCA channels through low-rank matrix recovery because of its strong capability in dealing with badly corrupted matrices. The experimental results on both simulated and real data validated the effectiveness of the proposed method both visually and quantitatively.     

Deterministic and probabilistic approaches for tracking virus particles in time-lapse fluorescence microscopy image sequences

Modern developments in time-lapse fluorescence microscopy enable the observation of a variety of processes exhibited by viruses. The dynamic nature of these processes requires the tracking of viruses over time to explore spatial–temporal relationships. In this work, we developed deterministic and probabilistic approaches for multiple virus tracking in multi-channel fluorescence microscopy images. The deterministic approaches follow a traditional two-step paradigm comprising particle localization based on either the spot-enhancing filter or 2D Gaussian fitting, as well as motion correspondence based on a global nearest neighbor scheme. Our probabilistic approaches are based on particle filters. We describe approaches based on a mixture of particle filters and based on independent particle filters. For the latter, we have developed a penalization strategy that prevents the problem of filter coalescence (merging) in cases where objects lie in close proximity. A quantitative comparison based on synthetic image sequences is carried out to evaluate the performance of our approaches. In total, eight different tracking approaches have been evaluated. We have also applied these approaches to real microscopy images of HIV-1 particles and have compared the tracking results with ground truth obtained from manual tracking. It turns out that the probabilistic approaches based on independent particle filters are superior to the deterministic schemes as well as to the approaches based on a mixture of particle filters.     

Miniaturized modular-array fluorescence microscopy

Fluorescence live-cell imaging allows for continuous interrogation of cellular behaviors, and the recent development of portable live-cell imaging platforms has rapidly transformed conventional schemes with high adaptability, cost-effective functionalities and easy accessibility to cell-based assays. However, broader applications remain restrictive due to compatibility with conventional cell culture workflow and biochemical sensors, accessibility to up-right physiological imaging, or parallelization of data acquisition. Here, we introduce miniaturized modular-array fluorescence microscopy (MAM) for compact live-cell imaging in flexible formats. We advance the current miniscopy technology to devise an up-right modular architecture, each combining a gradient-index (GRIN) objective and individually-addressed illumination and acquisition components. Parallelization of an array of such modular devices allows for multi-site data acquisition in situ using conventional off-the-shelf cell chambers. Compared with existing methods, the device offers a high fluorescence sensitivity and efficiency, exquisite spatiotemporal resolution (∼3 µm and up to 60 Hz), a configuration compatible with conventional cell culture assays and physiological imaging, and an effective parallelization of data acquisition. The system has been demonstrated using various calibration and biological samples and experimental conditions, representing a promising solution to time-lapse in situ single-cell imaging and analysis.     

Patient-specific image denoising for ultra-low-dose CT-guided lung biopsies

Purpose

Low-dose CT screening of the lungs is becoming a reality, triggering many more CT-guided lung biopsies. During these biopsies, the patient is submitted to repeated guiding scans with substantial cumulated radiation dose. Extension of the dose reduction to the biopsy procedure is therefore necessary. We propose an image denoising algorithm that specifically addresses the setup of CT-guided lung biopsies. It minimizes radiation exposure while keeping the image quality appropriate for navigation to the target lesion.

Methods

A database of high-SNR CT patches is used to filter noisy pixels in a non-local means framework, while explicitly enforcing local spatial consistency in order to preserve fine image details and structures. The patch database may be created from a multi-patient set of high-SNR lung scans. Alternatively, the first scan, acquired at high-SNR right before the needle insertion, can provide a convenient patient-specific patch database.

Results

The proposed algorithm is compared to state-of-the-art denoising algorithms for a dataset of 43 real CT-guided biopsy scans. Ultra-low-dose scans were simulated by synthetic noise addition to the sinogram, equivalent to a 96% reduction in radiation dose. The feature similarity score for the proposed algorithm outperformed the compared methods for all the scans in the dataset. The benefit of the patient-specific patch database over the multi-patient one is demonstrated in terms of recovered contrast for a tiny porcine lung nodule, following denoising with both approaches.

Conclusions

The proposed method provides a promising approach to the denoising of ultra-low-dose CT-guided biopsy images.

     

自适应非局域均值滤波的超声图像去噪

目的 提出一种去除超声图像噪声的新方法。方法 对超声图像进行非局域搜索,找到相似的图像块进行加权平均,降低噪声。通过定义一个特征强度,区分斑点噪声和图像边界;然后将特征强度引入非局域滤波方法中,对平坦区域和边界进行自适应滤波。结果 本方法可有效去除斑点噪声,提高噪声图像的峰值信噪比(PSNR)和结构相似度指数(SSIM),优于常规方法。结论 自适应非局域均值滤波可有效去噪,并保护超声图像特征。     

Limited-angle tomographic phase microscopy utilizing confocal scanning fluorescence microscopy

We present a multimodal imaging technique, combining tomographic phase microscopy with limited angular projection range and number, and two-channel spinning-disk confocal scanning fluorescence microscopy. This technique allows high-accuracy 3D refractive index (RI) profiling of live cells in spite of the missing projections. The cellular outer shape and its interior organelles measured by the confocal fluorescence imaging not only specify the cell in molecular levels, but also provide the 3D distributions of the whole cell as well as its organelles. We take these additional 3D morphological details as constraints in Gerchberg-Papoulis-based optical diffraction tomography algorithm. We then obtain an accurate 3D RI tomogram, even with a sparse angular range having a small number of perspective projections, otherwise providing low-accuracy RI reconstruction. Then, we obtain both cellular molecular specificity and inner RI values of the cell and its organelles. We compare the reconstructed 3D RI profiles of various samples, demonstrating the superiority of the proposed technique.     

Real-time OCT image denoising using a self-fusion neural network

Optical coherence tomography (OCT) has become the gold standard for ophthalmic diagnostic imaging. However, clinical OCT image-quality is highly variable and limited visualization can introduce errors in the quantitative analysis of anatomic and pathologic features-of-interest. Frame-averaging is a standard method for improving image-quality, however, frame-averaging in the presence of bulk-motion can degrade lateral resolution and prolongs total acquisition time. We recently introduced a method called self-fusion, which reduces speckle noise and enhances OCT signal-to-noise ratio (SNR) by using similarity between from adjacent frames and is more robust to motion-artifacts than frame-averaging. However, since self-fusion is based on deformable registration, it is computationally expensive. In this study a convolutional neural network was implemented to offset the computational overhead of self-fusion and perform OCT denoising in real-time. The self-fusion network was pretrained to fuse 3 frames to achieve near video-rate frame-rates. Our results showed a clear gain in peak SNR in the self-fused images over both the raw and frame-averaged OCT B-scans. This approach delivers a fast and robust OCT denoising alternative to frame-averaging without the need for repeated image acquisition. Real-time self-fusion image enhancement will enable improved localization of OCT field-of-view relative to features-of-interest and improved sensitivity for anatomic features of disease.     

基于非冗余平移不变小波变换的磁共振图像降噪

背景:MRI成像机制决定了其时间/空间分辨率和信噪比之间存在矛盾,因此图像降噪变得十分必要.目前基于离散小波变换的降噪方法广泛应用,然而存在平移敏感性的缺陷.目前已出现了克服平移敏感性的离散小波变换,但其冗余性导致计算复杂度的快速增加.目的:针对图像降噪设计小波滤波器,减小降采样过程的影响,保持离散小波变换的非冗余性,并针对MRI图像Rician噪声的降噪进行分析.方法:由于平移敏感性主要是由于离散小波变换分解时降采样产生的混叠项带来的,在保证非冗余的前提下,提出了通过减小混叠项的影响来减小平移敏感性.在此基础上,设计了一个双正交小波.最后,将其以常见的阈值降噪方法应用到磁共振图像Rician噪声的降噪中.结果与结论:文章提出了设计小波滤波器的新方法,即满足严格重构条件外满足一些附加要求,最后将设计过程简化为一个有约束条件的最优化过程.将设计的双正交小波应用于MR图像,仿真结果表明降噪效果较通常小波有所改善,间接表明了设计思路和方法的有效性.     

Frequency domain photoacoustic and fluorescence microscopy

We report on simultaneous frequency domain optical-resolution photoacoustic and fluorescence microscopy with sub-µm lateral resolution. With the help of a blood smear, we show that photoacoustic and fluorescence images provide complementary information. Furthermore, we compare theoretically predicted signal-to-noise ratios of sinusoidal modulation in frequency domain with pulsed excitation in time domain.OCIS codes: (170.5120) Photoacoustic imaging, (180.2520) Fluorescence microscopy, (110.5125) Photoacoustics, (170.3880) Medical and biological imaging, (140.2020) Diode lasers