Deep learning extended depth-of-field microscope for fast and slide-free histology

Microscopic evaluation of resected tissue plays a central role in the surgical management of cancer. Because optical microscopes have a limited depth-of-field (DOF), resected tissue is either frozen or preserved with chemical fixatives, sliced into thin sections placed on microscope slides, stained, and imaged to determine whether surgical margins are free of tumor cells—a costly and time- and labor-intensive procedure. Here, we introduce a deep-learning extended DOF (DeepDOF) microscope to quickly image large areas of freshly resected tissue to provide histologic-quality images of surgical margins without physical sectioning. The DeepDOF microscope consists of a conventional fluorescence microscope with the simple addition of an inexpensive (less than $10) phase mask inserted in the pupil plane to encode the light field and enhance the depth-invariance of the point-spread function. When used with a jointly optimized image-reconstruction algorithm, diffraction-limited optical performance to resolve subcellular features can be maintained while significantly extending the DOF (200 µm). Data from resected oral surgical specimens show that the DeepDOF microscope can consistently visualize nuclear morphology and other important diagnostic features across highly irregular resected tissue surfaces without serial refocusing. With the capability to quickly scan intact samples with subcellular detail, the DeepDOF microscope can improve tissue sampling during intraoperative tumor-margin assessment, while offering an affordable tool to provide histological information from resected tissue specimens in resource-limited settings.

Histopathology, or microscopic examination of thinly sectioned and stained tissue slices on glass slides, is the gold standard to diagnose and guide surgical management of cancer. To prepare histopathology slides, biopsies or surgical specimens are typically formalin-fixed and paraffin-embedded (FFPE), sliced with a microtome to around 5 µm, stained with hematoxylin and eosin (H&E) dyes, and evaluated under a light microscope. For intraoperative assessment, resected surgical specimens will be first cut with a scalpel into 3- to 4-mm-thick slices to access potential tumor margins on cross-sectional surfaces (Fig. 1A); the thick slices can then be quickly frozen to acquire thin (∼5 µm) transverse tissue sections for staining and microscopic examination. While frozen sections can reduce the processing time, a cryostat microtome is required to cut thin sections of frozen tissue, which still must be fixed and stained. Despite the central role of histopathology in cancer diagnosis, the time- and labor-intensive sample-preparation steps require specialized personnel and expensive equipment, while allowing for only limited sampling of resected tissue. In addition, these destructive procedures are susceptible to tissue-processing artifacts (1, 2) and can also interfere with downstream molecular or genetic analysis.Open in a separate windowFig. 1.DeepDOF microscope schematic and imaging performance in comparison to conventional microscopes for fluorescence imaging of intact tissue specimens. (A) Prior to imaging, the resected specimen is bread-loafed by using a pathology scalpel, and the cross-section surface can be evaluated for tumor-margin assessment. (B) Variations in the surface topology of intact tissue specimens exceed the DOF of a conventional microscope with subcellular resolution. In comparison, with the simple addition of an inexpensive phase mask, the end-to-end optimized DeepDOF microscope allows subcellular imaging of large areas of intact tissue samples at 5.4 cm²/min. (C) Based on a standard 4× objective (obj), the DeepDOF microscope combines wavefront encoding with deep-learning-enabled image reconstruction to significantly improve the DOF and, thus, the volumetric FOV while maintaining subcellular resolution. As a result, the DeepDOF microscope offers fast scanning of the cross-sectional surface of tissue slices without need for refocusing.In view of the challenges associated with standard histopathology, the ability to image cross-sectional surfaces of thick tissue slices (Fig. 1A) directly and nondestructively is highly desired. Recent studies have demonstrated successful imaging of large areas of intact specimens using fluorescence microscopy, including approaches based on confocal scanning (3, 4), structured illumination (5), and ultraviolet (UV) excitation (6). Clinical application of these techniques, however, is largely hindered by the shallow depth-of-field (DOF). In conventional microscopy, DOF is fundamentally coupled to lateral resolution:DOF∝λNA2∝resolution2λ.[1]As shown in Fig. 1C, in conventional microscopy with standard objectives, achieving subcellular lateral resolution (∼2 to 3 µm) restricts the DOF to ∼30 µm. This is almost one order of magnitude smaller than that needed to accommodate the variations in surface topography of freshly resected tissue surfaces, which can extend to up to 200 µm (7). As an example, Fig. 1 B, Upper Right shows a fluorescence image of an ex vivo porcine esophageal sample, stained with proflavine to highlight epithelial cell nuclei. In the image acquired with a conventional fluorescence microscope, the resulting defocus blur compromises the ability to visualize detailed cellular structures across a large field of view (FOV) without serial refocusing.To overcome the intrinsic optical constraints of conventional fluorescence microscopy for extended DOF, different approaches have been employed, such as decoupled illumination and detection in light-sheet microscopy (7), dynamic remote focusing (8, 9), and spatial and spectral multiplexing (10, 11); nonetheless, they usually require customized and expensive optics or complicated geometrical configurations. Alternatively, reflectance-based label-free modalities, including reflectance confocal microscopy and full-field optical coherence tomography, have been demonstrated for cancer-lesion characterization in skin and different types of epithelium (12–15). While initial results are promising, these systems are significantly more expensive (more than $100,000) than conventional microscopes due to their optomechanical complexity (16, 17). Computationally, extended DOF has also been demonstrated by using Fourier ptychographic microscopy (18), but the image reconstruction assumes a thin sample target transilluminated with oblique plane waves and is not suited for clinical fluorescence imaging.Wavefront encoding, when combined with computational methods, offers a convenient and inexpensive route to enhance imaging performance (19, 20). Wavefront modulating elements, such as cubic phase masks, annular phase masks, and other adaptive optics components, have been employed in photography, microscopy, and optical coherent microscopy to extend the DOF and to correct other forms of aberrations (21–32). Despite their adoption in different modalities, phase masks usually cause image degradation, thus necessitating a separately designed reconstruction algorithm to retrieve original features. Recently, deep learning is emerging as a powerful tool to complement microscopy for analysis of complex microscopic data (33–36). In this work, we integrate a wave-optics model with deep learning to develop a physics-informed, end-to-end optimization framework for extended DOF. In contrast to conventional approaches, the deep-learning framework optimizes the phase-mask design with large realistic data, while codesigning the reconstruction algorithm. Using this data-driven approach, we design, optimize, and experimentally validate the deep-learning extended DOF microscope (DeepDOF microscope), a low-cost (less than $6,000) computational microscope for fast and slide-free histology of surgical specimens. The DeepDOF microscope consists of two key co-optimized components: the phase mask and the image-reconstruction algorithm (Fig. 1 B, Upper Left). As shown in Fig. 1 B, Lower Left, and C, by jointly optimizing the phase mask and reconstruction algorithm, the DOF of the DeepDOF microscope is significantly extended to 200 µm, accommodating for variations in surface topology of thick cross-sectional tissue slices. Thanks to its capability to map irregular surfaces in a high-volumetric FOV (6.9 mm³ in DeepDOF microscope vs. 1.2 mm³ in a conventional microscope) with subcellular resolution, the DeepDOF microscope can image large areas of bread-loafed tissue slices without refocusing. Importantly, this is achieved by using an inexpensive phase-modulating element (less than $10 at production volume of 500 masks) that does not sacrifice optical throughput, making the DeepDOF microscope design readily adaptable to image fluorophores with low brightness.Here, we describe key components of the end-to-end optimized DeepDOF microscope from initial numerical simulation, to optical design, to subsequent experimental validation. We first present simulated results to jointly design and optimize the DeepDOF microscope optics and algorithm using a deep neural network. We then report characterization of the optimized DeepDOF phase mask, with simulated and experimental data. Furthermore, imaging of resected surgical specimens from the oral cavity is provided to validate clinical performance. We show that, using the current economical sample stage, DeepDOF can scan large specimens at 5.4 cm²/min, offering a fast, easy-to-use, and inexpensive alternative to standard histopathology for assessment of intact biopsies and surgical specimens with cellular detail.