Image analysis and machine learning in digital pathology: Challenges and opportunities

With the rise in whole slide scanner technology, large numbers of tissue slides are being scanned and represented and archived digitally. While digital pathology has substantial implications for telepathology, second opinions, and education there are also huge research opportunities in image computing with this new source of “big data”. It is well known that there is fundamental prognostic data embedded in pathology images. The ability to mine “sub-visual” image features from digital pathology slide images, features that may not be visually discernible by a pathologist, offers the opportunity for better quantitative modeling of disease appearance and hence possibly improved prediction of disease aggressiveness and patient outcome. However the compelling opportunities in precision medicine offered by big digital pathology data come with their own set of computational challenges. Image analysis and computer assisted detection and diagnosis tools previously developed in the context of radiographic images are woefully inadequate to deal with the data density in high resolution digitized whole slide images. Additionally there has been recent substantial interest in combining and fusing radiologic imaging and proteomics and genomics based measurements with features extracted from digital pathology images for better prognostic prediction of disease aggressiveness and patient outcome. Again there is a paucity of powerful tools for combining disease specific features that manifest across multiple different length scales.The purpose of this review is to discuss developments in computational image analysis tools for predictive modeling of digital pathology images from a detection, segmentation, feature extraction, and tissue classification perspective. We discuss the emergence of new handcrafted feature approaches for improved predictive modeling of tissue appearance and also review the emergence of deep learning schemes for both object detection and tissue classification. We also briefly review some of the state of the art in fusion of radiology and pathology images and also combining digital pathology derived image measurements with molecular “omics” features for better predictive modeling. The review ends with a brief discussion of some of the technical and computational challenges to be overcome and reflects on future opportunities for the quantitation of histopathology.     

Rapid pseudo-H&E imaging using a fluorescence-inbuilt optical coherence microscopic imaging system

A technique using Linnik-based optical coherence microscopy (OCM), with built-in fluorescence microscopy (FM), is demonstrated here to describe cellular-level morphology for fresh porcine and biobank tissue specimens. The proposed method utilizes color-coding to generate digital pseudo-H&E (p-H&E) images. Using the same camera, colocalized FM images are merged with corresponding morphological OCM images using a 24-bit RGB composition process to generate position-matched p-H&E images. From receipt of dissected fresh tissue piece to generation of stitched images, the total processing time is <15 min for a 1-cm² specimen, which is on average two times faster than frozen-section H&E process for fatty or water-rich fresh tissue specimens. This technique was successfully used to scan human and animal fresh tissue pieces, demonstrating its applicability for both biobank and veterinary purposes. We provide an in-depth comparison between p-H&E and human frozen-section H&E images acquired from the same metastatic sentinel lymph node slice (∼10 µm thick), and show the differences, like elastic fibers of a tiny blood vessel and cytoplasm of tumor cells. This optical sectioning technique provides histopathologists with a convenient assessment method that outputs large-field H&E-like images of fresh tissue pieces without requiring any physical embedment.     

Digital photography: a primer for pathologists

The computer and the digital camera provide a unique means for improving hematology education, research, and patient service. High quality photographic images of gross specimens can be rapidly and conveniently acquired with a high-resolution digital camera, and specialized digital cameras have been developed for photomicroscopy. Digital cameras utilize charge-coupled devices (CCD) or Complementary Metal Oxide Semiconductor (CMOS) image sensors to measure light energy and additional circuitry to convert the measured information into a digital signal. Since digital cameras do not utilize photographic film, images are immediately available for incorporation into web sites or digital publications, printing, transfer to other individuals by email, or other applications. Several excellent digital still cameras are now available for less than 2,500 dollars that capture high quality images comprised of more than 6 megapixels. These images are essentially indistinguishable from conventional film images when viewed on a quality color monitor or printed on a quality color or black and white printer at sizes up to 11x14 inches. Several recent dedicated digital photomicroscopy cameras provide an ultrahigh quality image output of more than 12 megapixels and have low noise circuit designs permitting the direct capture of darkfield and fluorescence images.There are many applications of digital images of pathologic specimens. Since pathology is a visual science, the inclusion of quality digital images into lectures, teaching handouts, and electronic documents is essential. A few institutions have gone beyond the basic application of digital images to developing large electronic hematology atlases, animated, audio-enhanced learning experiences, multidisciplinary Internet conferences, and other innovative applications. Digital images of single microscopic fields (single frame images) are the most widely utilized in hematology education at this time, but single images of many adjacent microscopic fields can be stitched together to prepare "zoomable" panoramas that encompass a large part of a microscope slide and closely simulate observation through a real microscope. With further advances in computer speed and Internet streaming technology, the virtual microscope could easily replace the real microscope in pathology education. Later in this decade, interactive immersive computer experiences may completely revolutionize hematology education and make the conventional lecture and laboratory format obsolete. Patient care is enhanced by the transmission of digital images to other individuals for consultation and education, and by the inclusion of these images in patient care documents. In research laboratories, digital cameras are widely used to document experimental results and to obtain experimental data.     

Conditional GANs based system for fibrosis detection and quantification in Hematoxylin and Eosin whole slide images

Assessing the degree of liver fibrosis is fundamental for the management of patients with chronic liver disease, in liver transplants procedures, and in general liver disease research. The fibrosis stage is best assessed by histopathologic evaluation, and Masson’s Trichrome stain (MT) is the stain of choice for this task in many laboratories around the world. However, the most used stain in histopathology is Hematoxylin Eosin (HE) which is cheaper, has a faster turn-around time and is the primary stain routinely used for evaluation of liver specimens. In this paper, we propose a novel digital pathology system that accurately detects and quantifies the footprint of fibrous tissue in HE whole slide images (WSI). The proposed system produces virtual MT images from HE using a deep learning model that learns deep texture patterns associated with collagen fibers. The training pipeline is based on conditional generative adversarial networks (cGAN), which can achieve accurate pixel-level transformation. Our comprehensive training pipeline features an automatic WSI registration algorithm, which qualifies the HE/MT training slides for the cGAN model. Using liver specimens collected during liver transplantation procedures, we conducted a range of experiments to evaluate the detected footprint of selected anatomical features. Our evaluation includes both image similarity and semantic segmentation metrics. The proposed system achieved enhanced results in the experiments with significant improvement over the state-of-the-art CycleGAN learning style, and over direct prediction of fibrosis in HE without having the virtual MT step.     

Informatics tools for quality in anatomic pathology

This article reviews three informatics tools developed as part of an overall quality program in an anatomic pathology laboratory. These tools include a tracking monitor for analyzing the entire testing process through pre-analytic, analytic, and postanalytic phases; the use of digital imaging in quality control of immunohistochemistry in the analytic phase; and a results-reporting monitor for flow of postanalytic data to patient data repositories.     

科研型肝癌肝移植临床标本库的建立及意义

背景：建立规范化的肝癌肝移植临床样本库可为开展相关领域研究提供资源平台,是进行科学研究的先决条件。目的：初步建立科研型肝癌肝移植临床标本库,研究并制定规范化的肝癌肝移植标本采集、处理、保存操作流程,建立完善、全面的肝癌肝癌肝移植标本信息化管理系统。方法：鉴国内外标本库建立的标准化程序,制定肝癌肝移植标本库操作流程及质量控制体系,规范化收集、处理及保存肝癌肝移植受者的肿瘤组织及血液标本,同时收集良性肝病肝移植受者及健康供者的肝组织及血液标本作为研究对照。对所收集的标本及相关临床资料信息进行系统化管理。结果与结论：自2009年8月肝癌肝移植标本库收集有完整临床资料的501例肝移植供、受者组织及血液标本,其中肝癌标本203例,良性肝病标本214例,健康供肝标本84例。肝癌标本包括肿瘤组织、癌旁组织及远端非癌组织标本,共计1773份;随机抽取45例标本进行质量监测,显示所收集的标本质量较好,开发了规范化的标本信息资料计算机管理系统。实验初步建立了具有一定规模的科研型肝癌肝移植标本库,有利于提高标本的质量,可操作性较好。     

将质量要求融入信息系统,以提高检验前质量控制能力

目的为了提高检验分析前环节的质量,利用信息系统对标本流程中的实验室前过程的质量控制要点进行实时提示、查询与监控。方法改进现有的电子检验申请单模式,实时跟踪、控制标本采集及运送过程中各环节的时间节点,并将标本采集的质量要求及检验项目的相关临床知识要点融入到开单与执行医嘱过程。结果 (1)为临床提供个性化专科或专病种检验申请项目单,如"急诊检验申请项目单",将可供急诊检验的项目集中于此,全院统一,非此单中所开检验申请一律视为非急诊项目;(2)在医生开单界面,随时通过双击鼠标查看检验项目:名称、标本种类、检测方法、标本采集要求、参考区间、临床意义、干扰因素及检验报告出具时间等内容;(3)护士执行医嘱时,每条检验项目行一栏均显示此项目的抗凝管类别或标本杯的条码前缀、采集量、采集要求及检验报告出具时间等内容;(4)所有人员均可通过"标本状态查询"功能,实时了解标本采集、签发、签收、处理、报告发送、报告打印等系列执行状态信息;(5)护士可通过"标本签收、退回"功能,查询本病区退回标本的原因、时间及操作人员信息。结论将检验分析前环节的质量控制要求融入信息系统,不仅方便临床医护实时了解检验相关知识的要求,而且完善了实验室对临床医护的培训、沟通体系,受到检验人员及临床医护的高度认可。     

Quality control in cytopathology

Quality control in cytopathology involves specimen collection, preparation, screening, and final diagnosis. The measures to assure the optimal performance and to detect the malfunctions in these areas are discussed. Careful selection and application of proper procedures, proper training and continuing education of cytotechnologists and pathologists, and reasonable workload for screening cytotechnologists and important factors in quality assurance.     

Automated classification of brain tumor type in whole-slide digital pathology images using local representative tiles

Computerized analysis of digital pathology images offers the potential of improving clinical care (e.g. automated diagnosis) and catalyzing research (e.g. discovering disease subtypes). There are two key challenges thwarting computerized analysis of digital pathology images: first, whole slide pathology images are massive, making computerized analysis inefficient, and second, diverse tissue regions in whole slide images that are not directly relevant to the disease may mislead computerized diagnosis algorithms. We propose a method to overcome both of these challenges that utilizes a coarse-to-fine analysis of the localized characteristics in pathology images. An initial surveying stage analyzes the diversity of coarse regions in the whole slide image. This includes extraction of spatially localized features of shape, color and texture from tiled regions covering the slide. Dimensionality reduction of the features assesses the image diversity in the tiled regions and clustering creates representative groups. A second stage provides a detailed analysis of a single representative tile from each group. An Elastic Net classifier produces a diagnostic decision value for each representative tile. A weighted voting scheme aggregates the decision values from these tiles to obtain a diagnosis at the whole slide level. We evaluated our method by automatically classifying 302 brain cancer cases into two possible diagnoses (glioblastoma multiforme (N = 182) versus lower grade glioma (N = 120)) with an accuracy of 93.1 % (p << 0.001). We also evaluated our method in the dataset provided for the 2014 MICCAI Pathology Classification Challenge, in which our method, trained and tested using 5-fold cross validation, produced a classification accuracy of 100% (p << 0.001). Our method showed high stability and robustness to parameter variation, with accuracy varying between 95.5% and 100% when evaluated for a wide range of parameters. Our approach may be useful to automatically differentiate between the two cancer subtypes.     

虚拟切片技术在肾内科临床、科研及教学中的应用前景

虚拟切片技术能够将传统的病理切片转换为数字图像, 使其具备长期保存、容易获取等特点, 正逐渐替代玻璃切片和光学显微镜用于病理远程会诊、科研和教学, 但其在肾内科领域的应用尚未广泛开展。本文通过文献复习和实践经验, 对虚拟切片技术在肾内科临床、科研和教学中的应用前景进行阐述, 一方面强调其潜在价值, 另一方面对目前实际应用中存在的问题及挑战提出解决建议。     

Sicherheitsstandards bei der Verwendung muskuloskelettaler Allografts

The transplantation of musculoskeletal tissues is widely used in orthopedics, sports medicine, traumatology and reconstruction after extensive tumor resection. The main risk after transplantation is the potential transmission of clinically relevant pathogens. Tissue banks have therefore established a safety level approach for the manufacture of musculoskeletal tissue transplants which includes donor selection, laboratory testing, tissue procurement, tissue processing and quality assurance. In addition, procedures have been developed to protect the biological properties of the tissue and to guarantee a high microbiological safety. In Germany validated chemical and thermal inactivation procedures have been successfully applied. A correct quality assurance in accordance with the Pharmaceuticals and Active Substances Production Act (AMWHV) and Transplantation Act (TPG) for tissue regulation ensures that tissue banks work within the law.     

The virtual blood film

The computer and the digital camera offer unprecedented possibilities for improving hematology education, research, and patient service. Peripheral blood smear images of exceptional quality can be acquired rapidly and conveniently from the peripheral blood smear with a modern, high-resolution digital camera and a quality microscope. Digital cameras use CCD or CMOS image sensors to measure light energy and additional circuitry to convert the measured information into a digital signal. Because digital cameras do not use photographic film, images are immediately available for incorporation into web sites or digital publications, printing, transfer to other individuals by e-mail, or other applications. Several excellent consumer digital still cameras are now available for less than $1000 that capture high-quality images comprised of more than three megapixels. These images are essentially indistinguishable from conventional film images when viewed on a quality color monitor or printed on a quality color or black and white printer at sizes up to 8 x 10 in. Several recent dedicated digital photomicroscopy cameras provide an ultrahigh quality image output of more than 12 megapixels and have low noise circuit designs permitting the direct capture of darkfield and fluorescence images. There are many applications of digital images of peripheral blood smears. Because hematology is a visual science, the inclusion of quality digital images into lectures, teaching handouts, and electronic documents is essential. A few institutions have gone beyond the basic application of digital images to develop large electronic hematology atlases; animated, audio-enhanced learning experiences; multidisciplinary Internet conferences; and other innovative applications. Digital images of single microscopic fields (single-frame images) are the most widely used in hematology education at this time, but single images of many adjacent microscopic fields can be stitched together to prepare zoomable panoramas that encompass a large part of a microscope slide and closely stimulate observation through a real microscope. With further advances in computer speed and Internet streaming technology, the virtual microscope could easily replace the real microscope in pathology education. Interactive, immersive computer experiences may completely revolutionize hematology education and make the conventional lecture and laboratory format obsolete later in this decade. Patient care is enhanced by the transmission of digital images to other individuals for consultation and education, and by the inclusion of these images in patient care documents. In research laboratories, digital cameras are widely used to document experimental results and obtain experimental data.     

Quality assurance review: improving diagnostic accuracy of percutaneous needle biopsy

Adequate pathologic slide preparation of percutaneous fine-needle biopsy (PFNB) specimens is necessary to maximize sensitivity and yield a tissue diagnosis. We present a quality control review of our biopsy technique and results in 71 aspiration biopsies done in 59 patients. Immediate cytology slide preparation in the radiology suite enabled us to increase our diagnostic yield from 53% to 84%, and allowed us to preserve PFNB, rather than open biopsy, as the primary diagnostic modality. We recommend regular review of biopsy techniques to confirm adequate diagnostic yields of these procedures.     

Tissue microarrays: applications in genomic research

The widespread application of tissue microarrays in cancer research and the clinical pathology laboratory demonstrates a versatile and portable technology. The rapid integration of tissue microarrays into biomarker discovery and validation processes reflects the forward thinking of researchers who have pioneered the high-density tissue microarray. The precise arrangement of hundreds of archival clinical tissue samples into a composite tissue microarray block is now a proven method for the efficient and standardized analysis of molecular markers. With applications in cancer research, tissue microarrays are a valuable tool in validating candidate markers discovered in highly sensitive genome-wide microarray experiments. With applications in clinical pathology, tissue microarrays are used widely in immunohistochemistry quality control and quality assurance. The timeline of a biomarker implicated in prostate neoplasia, which was identified by complementary DNA expression profiling, validated by tissue microarrays and is now used as a prognostic immunohistochemistry marker, is reviewed. The tissue microarray format provides opportunities for digital imaging acquisition, image processing and database integration. Advances in digital imaging help to alleviate previous bottlenecks in the research pipeline, permit computer image scoring and convey telepathology opportunities for remote image analysis. The tissue microarray industry now includes public and private sectors with varying degrees of research utility and offers a range of potential tissue microarray applications in basic research, prognostic oncology and drug discovery.     

Real-time three-dimensional optical coherence tomography image-guided core-needle biopsy system

Advances in optical imaging modalities, such as optical coherence tomography (OCT), enable us to observe tissue microstructure at high resolution and in real time. Currently, core-needle biopsies are guided by external imaging modalities such as ultrasound imaging and x-ray computed tomography (CT) for breast and lung masses, respectively. These image-guided procedures are frequently limited by spatial resolution when using ultrasound imaging, or by temporal resolution (rapid real-time feedback capabilities) when using x-ray CT. One feasible approach is to perform OCT within small gauge needles to optically image tissue microstructure. However, to date, no system or core-needle device has been developed that incorporates both three-dimensional OCT imaging and tissue biopsy within the same needle for true OCT-guided core-needle biopsy. We have developed and demonstrate an integrated core-needle biopsy system that utilizes catheter-based 3-D OCT for real-time image-guidance for target tissue localization, imaging of tissue immediately prior to physical biopsy, and subsequent OCT imaging of the biopsied specimen for immediate assessment at the point-of-care. OCT images of biopsied ex vivo tumor specimens acquired during core-needle placement are correlated with corresponding histology, and computational visualization of arbitrary planes within the 3-D OCT volumes enables feedback on specimen tissue type and biopsy quality. These results demonstrate the potential for using real-time 3-D OCT for needle biopsy guidance by imaging within the needle and tissue during biopsy procedures.     

Developing an institutional cancer biorepository for personalized medicine

High quality human biospecimens, such as tissue, blood, cell derivatives, and associated patient clinical information, are key elements of a scientific infrastructure that supports discovery and identification of molecular biomarkers and diagnostic agents. The goal of most biorepositories is to collect, process, store, and distribute human biospecimen for use in basic, translational and clinical research. A biorepository serving as the central hub provides investigators with an invaluable resource with appropriately examined and characterized biospecimens with associated patient clinical information. Expertise in standardization, quality control, and information technology, and awareness of cutting edge research developments are generally required for biorepository development and management. The availability of low cost whole genome profiles of individual tumors has opened up new possibilities for personalized medicine to deliver the most appropriate treatments to individual patients with minimal toxicity. A biorepository in support of personalized medicine thus requires the highest standards of operation and adequate funding, training and certification. This review provides an overview of the development of an institutional cancer biorepository for clinical research and personalized medicine advancement.     

Single image super-resolution for whole slide image using convolutional neural networks and self-supervised color normalization

High-quality whole slide scanners used for animal and human pathology scanning are expensive and can produce massive datasets, which limits the access to and adoption of this technique. As a potential solution to these challenges, we present a deep learning-based approach making use of single image super-resolution (SISR) to reconstruct high-resolution histology images from low-resolution inputs. Such low-resolution images can easily be shared, require less storage, and can be acquired quickly using widely available low-cost slide scanners. The network consists of multi-scale fully convolutional networks capable of capturing hierarchical features. Conditional generative adversarial loss is incorporated to penalize blurriness in the output images. The network is trained using a progressive strategy where the scaling factor is sampled from a normal distribution with an increasing mean. The results are evaluated with quantitative metrics and are used in a clinical histopathology diagnosis procedure which shows that the SISR framework can be used to reconstruct high-resolution images with clinical level quality. We further propose a self-supervised color normalization method that can remove staining variation artifacts. Quantitative evaluations show that the SISR framework can generalize well on unseen data collected from other patient tissue cohorts by incorporating the color normalization method.     

The Emergency Medicine Specimen Bank: An Innovative Approach To Biobanking In Acute Care

The Emergency Medicine Specimen Bank (EMSB) was developed to facilitate precision medicine in acute care. The EMSB is a biorepository of clinical health data and biospecimens collected from all adult English‐ or Spanish‐speaking individuals who are able and willing to provide consent and are treated at the UCHealth–University of Colorado Hospital Emergency Department. The EMSB is the first acute care biobank that seeks to enroll all patients, with all conditions who present to the ED. Acute care biobanking presents many challenges that are unique to acute care settings such as providing informed consent in a uniquely stressful and fast‐paced environment and collecting, processing, and storing samples for tens of thousands of patients per year. Here, we describe the process by which the EMSB overcame these challenges and was integrated into clinical workflow allowing for operation 24 hours a day, 7 days a week at a reasonable cost. Other institutions can implement this template, further increasing the power of biobanking research to inform treatment strategies and interventions for common and uncommon phenotypes in acute care settings.     

Low-cost whole slide imaging system with single-shot autofocusing based on color-multiplexed illumination and deep learning

Recent research on whole slide imaging (WSI) has greatly promoted the development of digital pathology. However, accurate autofocusing is still the main challenge for WSI acquisition and automated digital microscope. To address this problem, this paper describes a low cost WSI system and proposes a fast, robust autofocusing method based on deep learning. We use a programmable LED array for sample illumination. Before the brightfield image acquisition, we turn on a red and a green LED, and capture a color-multiplexed image, which is fed into a neural network for defocus distance estimation. After the focus tracking process, we employ a low-cost DIY adaptor to digitally adjust the photographic lens instead of the mechanical stage to perform axial position adjustment, and acquire the in-focus image under brightfield illumination. To ensure the calculation speed and image quality, we build a network model based on a ‘light weight’ backbone network architecture-MobileNetV3. Since the color-multiplexed coherent illuminated images contain abundant information about the defocus orientation, the proposed method enables high performance of autofocusing. Experimental results show that the proposed method can accurately predict the defocus distance of various types of samples and has good generalization ability for new types of samples. In the case of using GPU, the processing time for autofocusing is less than 0.1 second for each field of view, indicating that our method can further speed up the acquisition of whole slide images.     

充气固定肺标本新制作方法的影像学探讨

目的探讨充气固定肺叶标本制作的新方法--气液双重支撑固定法,并与用Heitzman法制作的肺叶标本进行影像与病理对照研究.方法收集40个肺叶切除术后的新鲜肺叶标本,分两组, A组26个用气液双重支撑固定法制作的肺叶标本,与B组14个用Heitzman法制作的肺叶标本在相应的HRCT扫描图像、软线摄影图像、大体标本、横断大体标本切片、组织学切片进行肺叶标本质量比较分析.结果肺叶标本质量评价:A组,Ⅰ级 22例(84.6%),Ⅱ级 4例(15.4%).B组Ⅱ级 5例(35.7%),Ⅲ级 9例(64.3%),制作方法与充气固定肺叶标本的质量有明显的相关性(P<0.01),A组标本质量好于B组.结论气液双重支撑固定法对研究次级肺小叶等肺的细微解剖结构,更精确地进行肺HRCT与病理对照研究有重要意义.