Gated STED microscopy with time-gated single-photon avalanche diode

Stimulated emission depletion (STED) microscopy provides fluorescence imaging with sub-diffraction resolution. Experimentally demonstrated at the end of the 90s, STED microscopy has gained substantial momentum and impact only in the last few years. Indeed, advances in many fields improved its compatibility with everyday biological research. Among them, a fundamental step was represented by the introduction in a STED architecture of the time-gated detection, which greatly reduced the complexity of the implementation and the illumination intensity needed. However, the benefits of the time-gated detection came along with a reduction of the fluorescence signal forming the STED microscopy images. The maximization of the useful (within the time gate) photon flux is then an important aspect to obtain super-resolved images. Here we show that by using a fast-gated single-photon avalanche diode (SPAD), i.e. a detector able to rapidly (hundreds picoseconds) switch-on and -off can improve significantly the signal-to-noise ratio (SNR) of the gated STED image. In addition to an enhancement of the image SNR, the use of the fast-gated SPAD reduces also the system complexity. We demonstrate these abilities both on calibration and biological sample. The experiments were carried on a gated STED microscope based on a STED beam operating in continuous-wave (CW), although the fast-gated SPAD is fully compatible with gated STED implementations based on pulsed STED beams.OCIS codes: (170.6920) Time-resolved imaging, (180.0180) Microscopy, (180.2520) Fluorescence microscopy, (230.5160) Photodetectors     

Encoder-decoder deep learning network for simultaneous reconstruction of fluorescence yield and lifetime distributions

A time-domain fluorescence molecular tomography in reflective geometry (TD-rFMT) has been proposed to circumvent the penetration limit and reconstruct fluorescence distribution within a 2.5-cm depth regardless of the object size. In this paper, an end-to-end encoder-decoder network is proposed to further enhance the reconstruction performance of TD-rFMT. The network reconstructs both the fluorescence yield and lifetime distributions directly from the time-resolved fluorescent signals. According to the properties of TD-rFMT, proper noise was added to the simulation training data and a customized loss function was adopted for self-supervised and supervised joint training. Simulations and phantom experiments demonstrate that the proposed network can significantly improve the spatial resolution, positioning accuracy, and accuracy of lifetime values.     

Visualizing topical drug uptake with conventional fluorescence microscopy and deep learning

Mapping the uptake of topical drugs and quantifying dermal pharmacokinetics (PK) presents numerous challenges. Though high resolution and high precision methods such as mass spectrometry offer the means to quantify drug concentration in tissue, these tools are complex and often expensive, limiting their use in routine experiments. For the many topical drugs that are naturally fluorescent, tracking fluorescence emission can be a means to gather critical PK parameters. However, skin autofluorescence can often overwhelm drug fluorescence signatures. Here we demonstrate the combination of standard epi-fluorescence imaging with deep learning for the visualization and quantification of fluorescent drugs in human skin. By training a U-Net convolutional neural network on a dataset of annotated images, drug uptake from both high "infinite" dose and daily clinical dose regimens can be measured and quantified. This approach has the potential to simplify routine topical product development in the laboratory.     

Joint regression-classification deep learning framework for analyzing fluorescence lifetime images using NADH and FAD

In this paper, we develop a deep neural network based joint classification-regression approach to identify microglia, a resident central nervous system macrophage, in the brain using fluorescence lifetime imaging microscopy (FLIM) data. Microglia are responsible for several key aspects of brain development and neurodegenerative diseases. Accurate detection of microglia is key to understanding their role and function in the CNS, and has been studied extensively in recent years. In this paper, we propose a joint classification-regression scheme that can incorporate fluorescence lifetime data from two different autofluorescent metabolic co-enzymes, FAD and NADH, in the same model. This approach not only represents the lifetime data more accurately but also provides the classification engine a more diverse data source. Furthermore, the two components of model can be trained jointly which combines the strengths of the regression and classification methods. We demonstrate the efficacy of our method using datasets generated using mouse brain tissue which show that our joint learning model outperforms results on the coenzymes taken independently, providing an efficient way to classify microglia from other cells.     

Real-time,continuous, fluorescence sensing in a freely-moving subject with an implanted hybrid VCSEL/CMOS biosensor

Performance improvements in instrumentation for optical imaging have contributed greatly to molecular imaging in living subjects. In order to advance molecular imaging in freely moving, untethered subjects, we designed a miniature vertical-cavity surface-emitting laser (VCSEL)-based biosensor measuring 1cm³ and weighing 0.7g that accurately detects both fluorophore and tumor-targeted molecular probes in small animals. We integrated a critical enabling component, a complementary metal-oxide semiconductor (CMOS) read-out integrated circuit, which digitized the fluorescence signal to achieve autofluorescence-limited sensitivity. After surgical implantation of the lightweight sensor for two weeks, we obtained continuous and dynamic fluorophore measurements while the subject was un-anesthetized and mobile. The technology demonstrated here represents a critical step in the path toward untethered optical sensing using an integrated optoelectronic implant.OCIS codes: (170.3890) Medical optics instrumentation, (130.6010) Sensors, (230.5160) Photodetectors, (130.5990) Semiconductors, (140.2020) Diode lasers     

Dynamic coronary roadmapping via catheter tip tracking in X-ray fluoroscopy with deep learning based Bayesian filtering

Percutaneous coronary intervention (PCI) is typically performed with image guidance using X-ray angiograms in which coronary arteries are opacified with X-ray opaque contrast agents. Interventional cardiologists typically navigate instruments using non-contrast-enhanced fluoroscopic images, since higher use of contrast agents increases the risk of kidney failure. When using fluoroscopic images, the interventional cardiologist needs to rely on a mental anatomical reconstruction. This paper reports on the development of a novel dynamic coronary roadmapping approach for improving visual feedback and reducing contrast use during PCI. The approach compensates cardiac and respiratory induced vessel motion by ECG alignment and catheter tip tracking in X-ray fluoroscopy, respectively. In particular, for accurate and robust tracking of the catheter tip, we proposed a new deep learning based Bayesian filtering method that integrates the detection outcome of a convolutional neural network and the motion estimation between frames using a particle filtering framework. The proposed roadmapping and tracking approaches were validated on clinical X-ray images, achieving accurate performance on both catheter tip tracking and dynamic coronary roadmapping experiments. In addition, our approach runs in real-time on a computer with a single GPU and has the potential to be integrated into the clinical workflow of PCI procedures, providing cardiologists with visual guidance during interventions without the need of extra use of contrast agent.     

Lutetium-containing sinoporphyrin sodium: a water-soluble photosensitizer with balanced fluorescence and phosphorescence for ratiometric oxygen sensing

Development of a photosensitizer for ratiometric O₂ sensing is desirable for the precise treatment of cancer by photodynamic therapy. Herein, lutetium(iii)-containing sinoporphyrin sodium (Lu-DVDMS) was designed as a phosphorescent photosensitizer to balance phosphorescence and fluorescence emissions for ratiometric O₂ sensing. Lu-DVDMS exhibited high water solubility, chemical stability, photostability, photosensitivity, and singlet-oxygen quantum yield of 0.23 ± 0.06. The phosphorescence and fluorescence quantum yields of Lu-DVDMS were 0.33 and 0.32%, respectively. Compared with the phosphorescence-to-fluorescence ratio (R) of gadolinium-DVDMS (Gd-DVDMS), which was >10, that of Lu-DVDMS was ∼1, facilitating ratiometric O₂ sensing. The relatively weak phosphorescence-inducing effect of Lu(iii) owing to the absence of paramagnetism, as compared to Gd(iii), balanced the phosphorescence and fluorescence emissions of Lu-DVDMS. The fluctuation of R for Lu-DVDMS was approximately one-sixth of Gd-DVDMS, owing to the high signal-to-noise ratio simultaneously achieved for both phosphorescence and fluorescence emissions. The intensity and lifetime Stern–Volmer plots for Lu-DVDMS were 0.9840 + 0.0024[O₂] and 0.9517 + 0.0034[O₂], respectively ([O₂]: oxygen concentration). Fast response and recovery times (<2 min) were achieved. The precision of oxygen detection using Lu-DVDMS was better than 0.5 μM in the 0–400 μM oxygen detection range. Therefore, Lu-DVDMS is a potential phosphorescent photosensitizer for ratiometric O₂ sensing.

A lutetium(iii)-porphyrin was designed as a phosphorescent photosensitizer to balance phosphorescence and fluorescence emissions for ratiometric O₂ sensing.     

Emetine identified in urine by HPLC, with fluorescence and ultraviolet/diode array detection, in a patient with cardiomyopathy

A 15-year-old girl with a four-month history of cardiac failure from undetermined cause was admitted to the hospital with weakness, fatigue, and weight loss. During her hospitalization she was found to have abused diet aids, laxatives, and cathartics. Although an electrocardiogram revealed nonspecific T-wave abnormalities and laboratory studies showed supranormal enzyme test results for creatine kinase and lactate dehydrogenase, no definite explanation of the cardiomyopathy was forthcoming. Ipecac abuse leading to cardiomyopathy was suspected early in the hospitalization. HPLC analysis of a urine sample showed emetine, a principle component of ipecac, the presence of which was later confirmed by more-specific HPLC analysis with photodiode array detection.     

Distortion and instability compensation with deep learning for rotational scanning endoscopic optical coherence tomography

Optical Coherence Tomography (OCT) is increasingly used in endoluminal procedures since it provides high-speed and high resolution imaging. Distortion and instability of images obtained with a proximal scanning endoscopic OCT system are significant due to the motor rotation irregularity, the friction between the rotating probe and outer sheath and synchronization issues. On-line compensation of artefacts is essential to ensure image quality suitable for real-time assistance during diagnosis or minimally invasive treatment. In this paper, we propose a new online correction method to tackle both B-scan distortion, video stream shaking and drift problem of endoscopic OCT linked to A-line level image shifting. The proposed computational approach for OCT scanning video correction integrates a Convolutional Neural Network (CNN) to improve the estimation of azimuthal shifting of each A-line. To suppress the accumulative error of integral estimation we also introduce another CNN branch to estimate a dynamic overall orientation angle. We train the network with semi-synthetic OCT videos by intentionally adding rotational distortion into real OCT scanning images. The results show that networks trained on this semi-synthetic data generalize to stabilize real OCT videos, and the algorithm efficacy is demonstrated on both ex vivo and in vivo data, where strong scanning artifacts are successfully corrected.     

A deep learning framework for pancreas segmentation with multi-atlas registration and 3D level-set

In this paper, we propose and validate a deep learning framework that incorporates both multi-atlas registration and level-set for segmenting pancreas from CT volume images. The proposed segmentation pipeline consists of three stages, namely coarse, fine, and refine stages. Firstly, a coarse segmentation is obtained through multi-atlas based 3D diffeomorphic registration and fusion. After that, to learn the connection feature, a 3D patch-based convolutional neural network (CNN) and three 2D slice-based CNNs are jointly used to predict a fine segmentation based on a bounding box determined from the coarse segmentation. Finally, a 3D level-set method is used, with the fine segmentation being one of its constraints, to integrate information of the original image and the CNN-derived probability map to achieve a refine segmentation. In other words, we jointly utilize global 3D location information (registration), contextual information (patch-based 3D CNN), shape information (slice-based 2.5D CNN) and edge information (3D level-set) in the proposed framework. These components form our cascaded coarse-fine-refine segmentation framework. We test the proposed framework on three different datasets with varying intensity ranges obtained from different resources, respectively containing 36, 82 and 281 CT volume images. In each dataset, we achieve an average Dice score over 82%, being superior or comparable to other existing state-of-the-art pancreas segmentation algorithms.     

Periodic arrays of plasmonic crossed-bowtie nanostructures interspaced with plasmonic nanocrosses for highly sensitive LSPR based chemical and biological sensing

In this paper, we present novel localized surface plasmon resonance (LSPR) sensors based on periodic arrays of gold crossed-bowtie nanostructures interspaced with gold nanocross pillars. Finite difference time domain (FDTD) numerical simulations were carried out to model bulk sensors as well as localized sensors based on the plasmonic nanostructures being proposed. The geometrical parameters of the plasmonic nanostructures are varied to obtain the best possible sensing performance in terms of sensitivity and figure of merit. A very high bulk sensitivity of 1753 nm per unit change in refractive index (nm RIU⁻¹), with a figure of merit for bulk sensing (FOM_bulk) of 3.65 RIU⁻¹, is obtained for these plasmonic nanostructures. This value of bulk sensitivity is higher in comparison to previously proposed LSPR sensors based on plasmonic nanopillars and nanocrosses. Moreover, the optimized LSPR sensors being proposed in this paper provide maximum sensitivity of localized refractive index sensing of 70 nm/nm with a FOM_localized of 0.33 nm⁻¹. This sensitivity of localized refractive index sensing is the highest reported thus far in comparison with previously reported LSPR sensors. It is also demonstrated that the operating resonance wavelengths of these LSPR sensors can be controllably tuned for specific applications by changing the dimensions of the plasmonic nanostructures.

Plasmonic nanostructure with very high localized LSPR sensitivity around 1310 nm and 1550 nm communication wavelengths.     

Low-cost whole slide imaging system with single-shot autofocusing based on color-multiplexed illumination and deep learning

Recent research on whole slide imaging (WSI) has greatly promoted the development of digital pathology. However, accurate autofocusing is still the main challenge for WSI acquisition and automated digital microscope. To address this problem, this paper describes a low cost WSI system and proposes a fast, robust autofocusing method based on deep learning. We use a programmable LED array for sample illumination. Before the brightfield image acquisition, we turn on a red and a green LED, and capture a color-multiplexed image, which is fed into a neural network for defocus distance estimation. After the focus tracking process, we employ a low-cost DIY adaptor to digitally adjust the photographic lens instead of the mechanical stage to perform axial position adjustment, and acquire the in-focus image under brightfield illumination. To ensure the calculation speed and image quality, we build a network model based on a ‘light weight’ backbone network architecture-MobileNetV3. Since the color-multiplexed coherent illuminated images contain abundant information about the defocus orientation, the proposed method enables high performance of autofocusing. Experimental results show that the proposed method can accurately predict the defocus distance of various types of samples and has good generalization ability for new types of samples. In the case of using GPU, the processing time for autofocusing is less than 0.1 second for each field of view, indicating that our method can further speed up the acquisition of whole slide images.     

基于剪切波弹性成像和卷积神经网络建立深度学习模型预测肾脏病变

目的 基于剪切波弹性成像（SWE）量化参数和卷积神经网络建立深度学习（DL）模型预测肾脏病变。方法 采集94例肾脏病变患者（病例组）和109名健康人（对照组）的肾脏超声SWE量化参数。利用卷积神经网络建立DL模型,比较DL模型和支持向量机、随机森林模型预测肾脏病变的敏感度、特异度、准确率和曲线下面积（AUC）。结果 DL模型对预测肾脏病变的敏感度为90.48%,特异度为100%,准确率为95.12%,AUC为0.93;支持向量机模型的敏感度、特异度、准确率和AUC分别为80.74%、80.71%、80.98%、0.90,随机森林模型分别为82.22%、77.87%、80.33%和0.88。DL模型预测敏感度、特异度、准确率和AUC均高于支持向量机和随机森林模型,与支持向量机模型和随机森林模型预测肾脏病变差异均有统计学意义（P均<0.05）。结论 基于SWE量化参数和卷积神经网络的DL模型预测肾脏疾病性能良好,具有一定临床价值。     

Superficial white matter analysis: An efficient point-cloud-based deep learning framework with supervised contrastive learning for consistent tractography parcellation across populations and dMRI acquisitions

Diffusion MRI tractography is an advanced imaging technique that enables in vivo mapping of the brain’s white matter connections. White matter parcellation classifies tractography streamlines into clusters or anatomically meaningful tracts. It enables quantification and visualization of whole-brain tractography. Currently, most parcellation methods focus on the deep white matter (DWM), whereas fewer methods address the superficial white matter (SWM) due to its complexity. We propose a novel two-stage deep-learning-based framework, Superficial White Matter Analysis (SupWMA), that performs an efficient and consistent parcellation of 198 SWM clusters from whole-brain tractography. A point-cloud-based network is adapted to our SWM parcellation task, and supervised contrastive learning enables more discriminative representations between plausible streamlines and outliers for SWM. We train our model on a large-scale tractography dataset including streamline samples from labeled long- and medium-range (over 40 mm) SWM clusters and anatomically implausible streamline samples, and we perform testing on six independently acquired datasets of different ages and health conditions (including neonates and patients with space-occupying brain tumors). Compared to several state-of-the-art methods, SupWMA obtains highly consistent and accurate SWM parcellation results on all datasets, showing good generalization across the lifespan in health and disease. In addition, the computational speed of SupWMA is much faster than other methods.     

A collaborative computer aided diagnosis (C-CAD) system with eye-tracking,sparse attentional model,and deep learning

Computer aided diagnosis (CAD) tools help radiologists to reduce diagnostic errors such as missing tumors and misdiagnosis. Vision researchers have been analyzing behaviors of radiologists during screening to understand how and why they miss tumors or misdiagnose. In this regard, eye-trackers have been instrumental in understanding visual search processes of radiologists. However, most relevant studies in this aspect are not compatible with realistic radiology reading rooms. In this study, we aim to develop a paradigm shifting CAD system, called collaborative CAD (C-CAD), that unifies CAD and eye-tracking systems in realistic radiology room settings. We first developed an eye-tracking interface providing radiologists with a real radiology reading room experience. Second, we propose a novel algorithm that unifies eye-tracking data and a CAD system. Specifically, we present a new graph based clustering and sparsification algorithm to transform eye-tracking data (gaze) into a graph model to interpret gaze patterns quantitatively and qualitatively. The proposed C-CAD collaborates with radiologists via eye-tracking technology and helps them to improve their diagnostic decisions. The C-CAD uses radiologists’ search efficiency by processing their gaze patterns. Furthermore, the C-CAD incorporates a deep learning algorithm in a newly designed multi-task learning platform to segment and diagnose suspicious areas simultaneously. The proposed C-CAD system has been tested in a lung cancer screening experiment with multiple radiologists, reading low dose chest CTs. Promising results support the efficiency, accuracy and applicability of the proposed C-CAD system in a real radiology room setting. We have also shown that our framework is generalizable to more complex applications such as prostate cancer screening with multi-parametric magnetic resonance imaging (mp-MRI).     

Dense biased networks with deep priori anatomy and hard region adaptation: Semi-supervised learning for fine renal artery segmentation

Fine renal artery segmentation on abdominal CT angiography (CTA) image is one of the most important tasks for kidney disease diagnosis and pre-operative planning. It will help clinicians locate each interlobar artery’s blood-feeding region via providing the complete 3D renal artery tree masks. However, it is still a task of great challenges due to the large intra-scale changes, large inter-anatomy variation, thin structures, small volume ratio and small labeled dataset of the fine renal artery. In this paper, we propose the first semi-supervised 3D fine renal artery segmentation framework, DPA-DenseBiasNet, which combines deep prior anatomy (DPA), dense biased network (DenseBiasNet) and hard region adaptation loss (HRA): 1) Based on our proposed dense biased connection, the DenseBiasNet fuses multi-receptive field and multi-resolution feature maps for large intra-scale changes. This dense biased connection also obtains a dense information flow and dense gradient flow so that the training is accelerated and the accuracy is enhanced. 2) DPA features extracted from an autoencoder (AE) are embedded in DenseBiasNet to cope with the challenge of large inter-anatomy variation and thin structures. The AE is pre-trained (unsupervised) by numerous unlabeled data to achieve the representation ability of anatomy features and these features are embedded in DenseBiasNet. This process will not introduce incorrect labels as optimization targets and thus contributes to a stable semi-supervised training strategy that is suitable for sensitive thin structures. 3) The HRA selects the loss value calculation region dynamically according to the segmentation quality so the network will pay attention to the hard regions in the training process and keep the class balanced.Experiments demonstrated that DPA-DenseBiasNet had high predictive accuracy and generalization with the Dice coefficient of 0.884 which increased by 0.083 compared with 3D U-Net (Çiçek et al., 2016). This revealed our framework with great potential for the 3D fine renal artery segmentation in clinical practice.