Microscopy illumination engineering using a low-cost liquid crystal display

Illumination engineering is critical for obtaining high-resolution, high-quality images in microscope settings. In a typical microscope, the condenser lens provides sample illumination that is uniform and free from glare. The associated condenser diaphragm can be manually adjusted to obtain the optimal illumination numerical aperture. In this paper, we report a programmable condenser lens for active illumination control. In our prototype setup, we used a $15 liquid crystal display as a transparent spatial light modulator and placed it at the back focal plane of the condenser lens. By setting different binary patterns on the display, we can actively control the illumination and the spatial coherence of the microscope platform. We demonstrated the use of such a simple scheme for multimodal imaging, including bright-field microscopy, darkfield microscopy, phase-contrast microscopy, polarization microscopy, 3D tomographic imaging, and super-resolution Fourier ptychographic imaging. The reported illumination engineering scheme is cost-effective and compatible with most existing platforms. It enables a turnkey solution with high flexibility for researchers in various communities. From the engineering point-of-view, the reported illumination scheme may also provide new insights for the development of multimodal microscopy and Fourier ptychographic imaging.OCIS codes: (170.2945) Illumination design, (170.0180) Microscopy, (170.3010) Image reconstruction techniques, (100.3190) Inverse problems     

Single full-FOV reconstruction Fourier ptychographic microscopy

Fourier ptychographic microscopy (FPM) is a recently developed computational imaging technique that has high-resolution and wide field-of-view (FOV). FPM bypasses the NA limit of the system by stitching a number of variable-illuminated measured images in Fourier space. On the basis of the wide FOV of the low NA objective, the high-resolution image with a wide FOV can be reconstructed through the phase recovery algorithm. However, the high-resolution reconstruction images are affected by the LED array point light source. The results are: (1) the intensities collected by the sample are severely declined when edge LEDs illuminate the sample; (2) the multiple reconstructions are caused by wavevectors inconsistency for the full FOV images. Here, we propose a new lighting scheme termed full FOV Fourier ptychographic microscopy (F³PM). By combining the LED array and telecentric lens, the method can provide plane waves with different angles while maintaining uniform intensity. Benefiting from the telecentric performance and f‒θ property of the telecentric lens, the system stability is improved and the relationship between the position of LED and its illumination angle is simplified. The excellent plane wave provided by the telecentric lens guarantees the same wavevector in the full FOV, and we use this wavevector to reconstruct the full FOV during one time. The area and diameter of the single reconstruction FOV reached 14.6mm² and 5.4 mm, respectively, and the diameter is very close to the field number (5.5 mm) of the 4× objective. Compared with the traditional FPM, we have increased the diameter of FOV in a single reconstruction by ∼ 10 times, eliminating the complicated steps of computational redundancy and image stitching.     

Spectral multiplexing and coherent-state decomposition in Fourier ptychographic imaging

Information multiplexing is important for biomedical imaging and chemical sensing. In this paper, we report a microscopy imaging technique, termed state-multiplexed Fourier ptychography (FP), for information multiplexing and coherent-state decomposition. Similar to a typical Fourier ptychographic setting, we use an array of light sources to illuminate the sample from different incident angles and acquire corresponding low-resolution images using a monochromatic camera. In the reported technique, however, multiple light sources are lit up simultaneously for information multiplexing, and the acquired images thus represent incoherent summations of the sample transmission profiles corresponding to different coherent states. We show that, by using the state-multiplexed FP recovery routine, we can decompose the incoherent mixture of the FP acquisitions to recover a high-resolution sample image. We also show that, color-multiplexed imaging can be performed by simultaneously turning on R/G/B LEDs for data acquisition. The reported technique may provide a solution for handling the partially coherent effect of light sources used in Fourier ptychographic imaging platforms. It can also be used to replace spectral filter, gratings or other optical components for spectral multiplexing and demultiplexing. With the availability of cost-effective broadband LEDs, the reported technique may open up exciting opportunities for computational multispectral imaging.OCIS codes: (170.3010) Image reconstruction techniques, (110.4234) Multispectral and hyperspectral imaging, (100.3190) Inverse problems, (170.0180) Microscopy     

Camera array based light field microscopy

This paper proposes a novel approach for high-resolution light field microscopy imaging by using a camera array. In this approach, we apply a two-stage relay system for expanding the aperture plane of the microscope into the size of an imaging lens array, and utilize a sensor array for acquiring different sub-apertures images formed by corresponding imaging lenses. By combining the rectified and synchronized images from 5 × 5 viewpoints with our prototype system, we successfully recovered color light field videos for various fast-moving microscopic specimens with a spatial resolution of 0.79 megapixels at 30 frames per second, corresponding to an unprecedented data throughput of 562.5 MB/s for light field microscopy. We also demonstrated the use of the reported platform for different applications, including post-capture refocusing, phase reconstruction, 3D imaging, and optical metrology.OCIS codes: (180.6900) Three-dimensional microscopy, (110.1758) Computational imaging, (170.0110) Imaging systems, (100.3010) Image reconstruction techniques     

Wide field-of-view fluorescence image deconvolution with aberration-estimation from Fourier ptychography

This paper presents a method to simultaneously acquire an aberration-corrected, wide field-of-view fluorescence image and a high-resolution coherent bright-field image using a computational microscopy method. First, the procedure applies Fourier ptychographic microscopy (FPM) to retrieve the amplitude and phase of a sample, at a resolution that significantly exceeds the cutoff spatial frequency of the microscope objective lens. At the same time, redundancy within the set of acquired FPM bright-field images offers a means to estimate microscope aberrations. Second, the procedure acquires an aberrated fluorescence image, and computationally improves its resolution through deconvolution with the estimated aberration map. An experimental demonstration successfully improves the bright-field resolution of fixed, stained and fluorescently tagged HeLa cells by a factor of 4.9, and reduces the error caused by aberrations in a fluorescence image by up to 31%, over a field of view of 6.2 mm by 9.3 mm. For optimal deconvolution, we show the fluorescence image needs to have a signal-to-noise ratio of at least ~18.OCIS codes: (180.2520) Fluorescence microscopy, (070.0070) Fourier optics and signal processing     

Robust Fourier ptychographic microscopy via a physics-based defocusing strategy for calibrating angle-varied LED illumination

Fourier ptychographic microscopy (FPM) is a recently developed computational imaging technique for wide-field, high-resolution microscopy with a high space-bandwidth product. It integrates the concepts of synthetic aperture and phase retrieval to surpass the resolution limit imposed by the employed objective lens. In the FPM framework, the position of each sub-spectrum needs to be accurately known to ensure the success of the phase retrieval process. Different from the conventional methods with mechanical adjustment or data-driven optimization strategies, here we report a physics-based defocusing strategy for correcting large-scale positional deviation of the LED illumination in FPM. Based on a subpixel image registration process with a defocused object, we can directly infer the illumination parameters including the lateral offsets of the light source, the in-plane rotation angle of the LED array, and the distance between the sample and the LED board. The feasibility and effectiveness of our method are validated with both simulations and experiments. We show that the reported strategy can obtain high-quality reconstructions of both the complex object and pupil function even the LED array is randomly placed under the sample with both unknown lateral offsets and rotations. As such, it enables the development of robust FPM systems by reducing the requirements on fine mechanical adjustment and data-driven correction in the construction process.     

Resolution doubling with a reduced number of image acquisitions

Structured illumination technique enhances the lateral resolution by projecting non-uniform intensity patterns on a sample. In a typical implementation, three lateral phase shifts (0, 2π/3, 4π/3) are needed for each orientation of the sinusoidal pattern, and 3 different orientations are needed to double the bandwidth isotopically in the Fourier domain. To this end, 9 incoherent images are needed in the acquisition process. In this paper, we discuss an imaging strategy for the structured illumination technique and demonstrate the use of a modified incoherent Fourier ptychographic procedure for reducing the number of acquisitions. In the first implementation, we used complementary sinusoidal patterns for sample illumination. We show that, the number of lateral phase shifts can be reduced from 3 to 2 for each orientation of the sinusoidal pattern and the total number of image acquisitions can be reduced to 6 with 3 orientations. In the second implementation, we further reduce the number of image acquisitions to 4. We also show that, the resolution-doubled image can be recovered even with unknown phases of the sinusoidal patterns. We validate the proposed imaging procedure with non-fluorescence samples. The reported approach may shorten the acquisition time of super-resolution imaging and reduce phototoxicity of biological samples.OCIS codes: (170.3010) Image reconstruction techniques, (170.2945) Illumination design, (170.0180) Microscopy     

Digital photography: a primer for pathologists

The computer and the digital camera provide a unique means for improving hematology education, research, and patient service. High quality photographic images of gross specimens can be rapidly and conveniently acquired with a high-resolution digital camera, and specialized digital cameras have been developed for photomicroscopy. Digital cameras utilize charge-coupled devices (CCD) or Complementary Metal Oxide Semiconductor (CMOS) image sensors to measure light energy and additional circuitry to convert the measured information into a digital signal. Since digital cameras do not utilize photographic film, images are immediately available for incorporation into web sites or digital publications, printing, transfer to other individuals by email, or other applications. Several excellent digital still cameras are now available for less than 2,500 dollars that capture high quality images comprised of more than 6 megapixels. These images are essentially indistinguishable from conventional film images when viewed on a quality color monitor or printed on a quality color or black and white printer at sizes up to 11x14 inches. Several recent dedicated digital photomicroscopy cameras provide an ultrahigh quality image output of more than 12 megapixels and have low noise circuit designs permitting the direct capture of darkfield and fluorescence images.There are many applications of digital images of pathologic specimens. Since pathology is a visual science, the inclusion of quality digital images into lectures, teaching handouts, and electronic documents is essential. A few institutions have gone beyond the basic application of digital images to developing large electronic hematology atlases, animated, audio-enhanced learning experiences, multidisciplinary Internet conferences, and other innovative applications. Digital images of single microscopic fields (single frame images) are the most widely utilized in hematology education at this time, but single images of many adjacent microscopic fields can be stitched together to prepare "zoomable" panoramas that encompass a large part of a microscope slide and closely simulate observation through a real microscope. With further advances in computer speed and Internet streaming technology, the virtual microscope could easily replace the real microscope in pathology education. Later in this decade, interactive immersive computer experiences may completely revolutionize hematology education and make the conventional lecture and laboratory format obsolete. Patient care is enhanced by the transmission of digital images to other individuals for consultation and education, and by the inclusion of these images in patient care documents. In research laboratories, digital cameras are widely used to document experimental results and to obtain experimental data.     

Wide field intravital imaging by two-photon-excitation digital-scanned light-sheet microscopy (2p-DSLM) with a high-pulse energy laser

Digital-scanned light-sheet microscopy (DSLM) illuminates a sample in a plane and captures single-photon–excitation fluorescence images with a camera from a direction perpendicular to the light sheet. This method is potentially useful for observing biological specimens, because image acquisition is relatively fast, resulting in reduction of phototoxicity. However, DSLM cannot be effectively applied to high-scattering materials due to the image blur resulting from thickening of the light sheet by scattered photons. However, two-photon–excitation DSLM (2p-DSLM) enables collection of high-contrast image with near infrared (NIR) excitation. In conventional 2p-DSLM, the minimal excitation volume for two-photon excitation restricts the field of view. In this study, we achieved wide-field 2p-DSLM by using a high–pulse energy fiber laser, and then used this technique to perform intravital imaging of a small model fish species, medaka (Oryzias latipes). Wide fields of view (>700 μm) were achieved by using a low–numerical aperture (NA) objective lens and high–peak energy NIR excitation at 1040 nm. We also performed high-speed imaging at near-video rate and successfully captured the heartbeat movements of a living medaka fish at 20 frames/sec.OCIS codes: (180.0180) Microscopy, (000.1430) Biology and medicine, (180.2520) Fluorescence microscopy, (180.4315) Nonlinear microscopy, (110.6880) Three-dimensional image acquisition     

High-resolution 3D Fourier ptychographic reconstruction using a hemispherical illumination source with multiplexed-coded strategy

Fourier ptychography is a promising and flexible imaging technique that can achieve 2D quantitative reconstruction with higher resolution beyond the limitation of the system. Meanwhile, by using different imaging models, the same platform can be applied to achieve 3D refractive index reconstruction. To improve the illumination NA as much as possible while reducing the intensity attenuation problem caused by the LED board used in the traditional FP platform, we apply a hemispherical lighting structure and design a new LED arrangement according to 3D Fourier diffraction theory. Therefore, we could obtain the illumination of 0.98NA using 187 LEDs and achieve imaging half-pitch resolutions of ∼174 nm and ∼524 nm for the lateral and axial directions respectively, using a 40×/0.6NA objective lens. Furthermore, to reduce the number of captured images required and realize real-time data collection, we apply the multiplexed-coded illumination strategy and compare several coded patterns through simulation and experiment. Through comparison, we determined a radial-coded illumination pattern that could achieve more similar results as sequential scanning and increase the acquisition speed to above 1 Hz. Therefore, this paper provides the possibility of this technique in real-time 3D observation of in vitro live samples.     

The virtual blood film

The computer and the digital camera offer unprecedented possibilities for improving hematology education, research, and patient service. Peripheral blood smear images of exceptional quality can be acquired rapidly and conveniently from the peripheral blood smear with a modern, high-resolution digital camera and a quality microscope. Digital cameras use CCD or CMOS image sensors to measure light energy and additional circuitry to convert the measured information into a digital signal. Because digital cameras do not use photographic film, images are immediately available for incorporation into web sites or digital publications, printing, transfer to other individuals by e-mail, or other applications. Several excellent consumer digital still cameras are now available for less than $1000 that capture high-quality images comprised of more than three megapixels. These images are essentially indistinguishable from conventional film images when viewed on a quality color monitor or printed on a quality color or black and white printer at sizes up to 8 x 10 in. Several recent dedicated digital photomicroscopy cameras provide an ultrahigh quality image output of more than 12 megapixels and have low noise circuit designs permitting the direct capture of darkfield and fluorescence images. There are many applications of digital images of peripheral blood smears. Because hematology is a visual science, the inclusion of quality digital images into lectures, teaching handouts, and electronic documents is essential. A few institutions have gone beyond the basic application of digital images to develop large electronic hematology atlases; animated, audio-enhanced learning experiences; multidisciplinary Internet conferences; and other innovative applications. Digital images of single microscopic fields (single-frame images) are the most widely used in hematology education at this time, but single images of many adjacent microscopic fields can be stitched together to prepare zoomable panoramas that encompass a large part of a microscope slide and closely stimulate observation through a real microscope. With further advances in computer speed and Internet streaming technology, the virtual microscope could easily replace the real microscope in pathology education. Interactive, immersive computer experiences may completely revolutionize hematology education and make the conventional lecture and laboratory format obsolete later in this decade. Patient care is enhanced by the transmission of digital images to other individuals for consultation and education, and by the inclusion of these images in patient care documents. In research laboratories, digital cameras are widely used to document experimental results and obtain experimental data.     

Tracking of a bronchoscope using epipolar geometry analysis and intensity-based image registration of real and virtual endoscopic images

This paper describes a method for tracking the camera motion of a flexible endoscope, in particular a bronchoscope, using epipolar geometry analysis and intensity-based image registration. The method proposed here does not use a positional sensor attached to the endoscope. Instead, it tracks camera motion using real endoscopic (RE) video images obtained at the time of the procedure and X-ray CT images acquired before the endoscopic examination. A virtual endoscope system (VES) is used for generating virtual endoscopic (VE) images. The basic idea of this tracking method is to find the viewpoint and view direction of the VES that maximizes a similarity measure between the VE and RE images. To assist the parameter search process, camera motion is also computed directly from epipolar geometry analysis of the RE video images. The complete method consists of two steps: (a) rough estimation using epipolar geometry analysis and (b) precise estimation using intensity-based image registration. In the rough registration process, the method computes camera motion from optical flow patterns between two consecutive RE video image frames using epipolar geometry analysis. In the image registration stage, we search for the VES viewing parameters that generate the VE image that is most similar to the current RE image. The correlation coefficient and the mean square intensity difference are used for measuring image similarity. The result obtained in the rough estimation process is used for restricting the parameter search area. We applied the method to bronchoscopic video image data from three patients who had chest CT images. The method successfully tracked camera motion for about 600 consecutive frames in the best case. Visual inspection suggests that the tracking is sufficiently accurate for clinical use. Tracking results obtained by performing the method without the epipolar geometry analysis step were substantially worse. Although the method required about 20 s to process one frame, the results demonstrate the potential of image-based tracking for use in an endoscope navigation system.     

Automated wide-field malaria parasite infection detection using Fourier ptychography on stain-free thin-smears

Diagnosis of malaria in endemic areas is hampered by the lack of a rapid, stain-free and sensitive method to directly identify parasites in peripheral blood. Herein, we report the use of Fourier ptychography to generate wide-field high-resolution quantitative phase images of erythrocytes infected with malaria parasites, from a whole blood sample. We are able to image thousands of erythrocytes (red blood cells) in a single field of view and make a determination of infection status of the quantitative phase image of each segmented cell based on machine learning (random forest) and deep learning (VGG16) models. Our random forest model makes use of morphology and texture based features of the quantitative phase images. In order to label the quantitative images of the cells as either infected or uninfected before training the models, we make use of a Plasmodium berghei strain expressing GFP (green fluorescent protein) in all life cycle stages. By overlaying the fluorescence image with the quantitative phase image we could identify the infected subpopulation of erythrocytes for labelling purposes. Our machine learning model (random forest) achieved 91% specificity and 72% sensitivity while our deep learning model (VGG16) achieved 98% specificity and 57% sensitivity. These results highlight the potential for quantitative phase imaging coupled with artificial intelligence to develop an easy to use platform for the rapid and sensitive diagnosis of malaria.     

Video-rate high-resolution single-pixel nonscanning photoacoustic microscopy

Optical-resolution photoacoustic microscopy (OR-PAM) is widely utilized in biomedical applications because of its ability to noninvasively image biological tissues in vivo while providing high-resolution morphological and functional information. However, one drawback of conventional OR-PAM is its imaging speed, which is restricted by the scanning technique employed. To achieve a higher imaging frame rate, we present video-rate high-resolution single-pixel nonscanning photoacoustic microscopy (SPN-PAM), which utilizes Fourier orthogonal basis structured planar illumination to overcome the above-mentioned limitations. A 473 × 473 µm² imaging field of view (FOV) with 3.73 µm lateral resolution and video-rate imaging of 30 Hz were achieved. In addition, in both in vitro cell and in vivo mouse vascular hemodynamic imaging experiments, high-quality images were obtained at ultralow sampling rates. Thus, the proposed high-resolution SPN-PAM with video-rate imaging speed provides new insights into high-speed PA imaging and could be a powerful tool for rapid biological imaging.     

Periscope for noninvasive two-photon imaging of murine retina in vivo

Two-photon microscopy allows visualization of subcellular structures in the living animal retina. In previously reported experiments it was necessary to apply a contact lens to each subject. Extending this technology to larger animals would require fitting a custom contact lens to each animal and cumbersome placement of the living animal head on microscope stage. Here we demonstrate a new device, periscope, for coupling light energy into mouse eye and capturing emitted fluorescence. Using this periscope we obtained images of the RPE and their subcellular organelles, retinosomes, with larger field of view than previously reported. This periscope provides an interface with a commercial microscope, does not require contact lens and its design could be modified to image retina in larger animals.OCIS codes: (170.0110) Imaging systems, (170.4460) Ophthalmic optics and devices, (170.5755) Retina scanning, (180.4315) Nonlinear microscopy, (330.7327) Visual optics, ophthalmic instrumentation     

Real-time imaging with a new ultrasonic camera: Part II,preliminary studies in normal adults

A new transmission ultrasound camera system has provided focused images in real time through the abdomen of adult volunteers at average incident sound intensity levels of less than 1 mW/cm². The camera uses a unique acousto-mechanical lens/deflection system and a linear array of piezoelectric transducers to detect the ultrasound image. Bony structures and costal cartilages were clearly seen. The colon was regularly seen and the kidney could be outlined in several subjects. The stomach and duodenal bulb were demonstrated and peristalsis was observed using a barium contrast material. The camera has the potential of extending the use of ultrasound in clinical medicine providing an ultrasound soft tissue imaging system analogous to an x-ray fluoroscope. The need for further clinical evaluation and research to improve the images is emphasized.     

Resolution-enhancement for an orthographic-view image display in an integral imaging microscope system

Due to the limitations of micro lens arrays and camera sensors, images on display devices through the integral imaging microscope systems have been suffering for a low-resolution. In this paper, a resolution-enhanced orthographic-view image display method for integral imaging microscopy is proposed and demonstrated. Iterative intermediate-view reconstructions are performed based on bilinear interpolation using neighborhood elemental image information, and a graphics processing unit parallel processing algorithm is applied for fast image processing. The proposed method is verified experimentally and the effective results are presented in this paper.OCIS codes: (180.6900) Three-dimensional microscopy, (100.6890) Three-dimensional image processing, (120.2040) Displays     

InstantScope: a low-cost whole slide imaging system with instant focal plane detection

We report the development of a high-throughput whole slide imaging (WSI) system by adapting a cost-effective optomechanical add-on kit to existing microscopes. Inspired by the phase detection concept in professional photography, we attached two pinhole-modulated cameras at the eyepiece ports for instant focal plane detection. By adjusting the positions of the pinholes, we can effectively change the view angle for the sample, and as such, we can use the translation shift of the two pinhole-modulated images to identify the optimal focal position. By using a small pinhole size, the focal-plane-detection range is on the order of millimeter, orders of magnitude longer than the objective’s depth of field. We also show that, by analyzing the phase correlation of the pinhole-modulated images, we can determine whether the sample contains one thin section, folded sections, or multiple layers separated by certain distances – an important piece of information prior to a detailed z scan. In order to achieve system automation, we deployed a low-cost programmable robotic arm to perform sample loading and $14 stepper motors to drive the microscope stage to perform x-y scanning. Using a 20X objective lens, we can acquire a 2 gigapixel image with 14 mm by 8 mm field of view in 90 seconds. The reported platform may find applications in biomedical research, telemedicine, and digital pathology. It may also provide new insights for the development of high-content screening instruments.OCIS codes: (170.0180) Microscopy, (170.3010) Image reconstruction techniques, (110.0110) Imaging systems, (110.1220) Apertures     

Scattered-light imaging in vivo tracks fast and slow processes of neurophysiological activation.

We imaged fast optical changes associated with evoked neural activation in the dorsal brainstem of anesthetized rats, using a novel imaging device. The imager consisted of a gradient-index (GRIN) lens, a microscope objective, and a miniature charged-coupled device (CCD) video camera. We placed the probe in contact with tissue above cardiorespiratory areas of the nucleus of the solitary tract and illuminated the tissue with 780-nm light through flexible fibers around the probe perimeter. The focus depth was adjusted by moving the camera and microscope objective relative to the fixed GRIN lens. Back-scattered light images were relayed through the GRIN lens to the CCD camera. Video frames were digitized at 100 frames per second, along with tracheal pressure, arterial blood pressure, and electrocardiogram signals recorded at 1 kHz per channel. A macroelectrode placed under the GRIN lens recorded field potentials from the imaged area. Aortic, vagal, and superior laryngeal nerves were dissected free of surrounding tissue within the neck. Separate shocks to each dissected nerve elicited evoked electrical responses and caused localized optical activity patterns. The optical response was modeled by four distinct temporal components corresponding to putative physical mechanisms underlying scattered light changes. Region-of-interest analysis revealed image areas which were dominated by one or more of the different time-course components, some of which were also optimally recorded at different tissue depths. Two slow optical components appear to correspond to hemodynamic responses to metabolic demand associated with activation. Two fast optical components paralleled electrical evoked responses.     

Quantitative Jones matrix imaging using vectorial Fourier ptychography

This paper presents a microscopic imaging technique that uses variable-angle illumination to recover the complex polarimetric properties of a specimen at high resolution and over a large field-of-view. The approach extends Fourier ptychography, which is a synthetic aperture-based imaging approach to improve resolution with phaseless measurements, to additionally account for the vectorial nature of light. After images are acquired using a standard microscope outfitted with an LED illumination array and two polarizers, our vectorial Fourier ptychography (vFP) algorithm solves for the complex 2x2 Jones matrix of the anisotropic specimen of interest at each resolved spatial location. We introduce a new sequential Gauss-Newton-based solver that additionally jointly estimates and removes polarization-dependent imaging system aberrations. We demonstrate effective vFP performance by generating large-area (29 mm²), high-resolution (1.24 μm full-pitch) reconstructions of sample absorption, phase, orientation, diattenuation, and retardance for a variety of calibration samples and biological specimens.