Characterization of Radiologists’ Search Strategies for Lung Nodule Detection: Slice-Based Versus Volumetric Displays

The goal of this study was to assess whether radiologists’ search paths for lung nodule detection in chest computed tomography (CT) between different rendering and display schemes have reliable properties that can be exploited as an indicator of ergonomic efficiency for the purpose of comparing different display paradigms. Eight radiologists retrospectively viewed 30 lung cancer screening CT exams, containing a total of 91 nodules, in each of three display modes [i.e., slice-by-slice, orthogonal maximum intensity projection (MIP) and stereoscopic] for the purpose of detecting and classifying lung nodules. Radiologists’ search patterns in the axial direction were recorded and analyzed along with the location, size, and shape for each detected feature, and the likelihood that the feature is an actual nodule. Nodule detection performance was analyzed by employing free-response receiver operating characteristic methods. Search paths were clearly different between slice-by-slice displays and volumetric displays but, aside from training and novelty effects, not between MIP and stereographic displays. Novelty and training effects were associated with the stereographic display mode, as evidenced by differences between the beginning and end of the study. The stereo display provided higher detection and classification performance with less interpretation time compared to other display modes tested in the study; however, the differences were not statistically significant. Our preliminary results indicate a potential role for the use of radiologists’ search paths in evaluating the relative ergonomic efficiencies of different display paradigms, but systematic training and practice is necessary to eliminate training curve and novelty effects before search strategies can be meaningfully compared.     

Preliminary Evaluation of Biplane Correlation (BCI) Stereographic Imaging for Lung Nodule Detection

A biplane correlation (BCI) imaging system obtains images that can be viewed in stereo, thereby minimizing overlapping structures. This study investigated whether using stereoscopic visualization provides superior lung nodule detection compared to standard postero-anterior (PA) image display. Images were acquired at two oblique views of ±3° as well as at a standard PA position from 60 patients. Images were processed using optimal parameters and displayed on a stereoscopic display. The PA image was viewed in the standard format, while the oblique views were paired to provide a stereoscopic view of the subject. A preliminary observer study was performed with four radiologists who viewed and scored the PA image then viewed and scored the BCI stereoscopic image. The BCI stereoscopic viewing of lung nodules resulted in 71 % sensitivity and 0.31 positive predictive value (PPV) index compared to PA results of 86 % sensitivity and 0.26 PPV index. The sensitivity for lung nodule detection with the BCI stereoscopic system was reduced by 15 %; however, the total number of false positives reported was reduced by 35 % resulting in an improved PPV index of 20 %. The preliminary results indicate observer dependency in terms of relative advantage of either system in the detection of lung nodules, but overall equivalency of the two methods with promising potential for BCI as an adjunct diagnostic technique.     

Deep Learning-based Artificial Intelligence Improves Accuracy of Error-prone Lung Nodules

Introduction: Early detection of lung cancer is one way to improve outcomes. Improving the detection of nodules on chest CT scans is important. Previous artificial intelligence (AI) modules show rapid advantages, which improves the performance of detecting lung nodules in some datasets. However, they have a high false-positive (FP) rate. Its effectiveness in clinical practice has not yet been fully proven. We aimed to use AI assistance in CT scans to decrease FP.Materials and methods: CT images of 60 patients were obtained. Five senior doctors who were blinded to these cases participated in this study for the detection of lung nodules. Two doctors performed manual detection and labeling of lung nodules without AI assistance. Another three doctors used AI assistance to detect and label lung nodules before manual interpretation. The AI program is based on a deep learning framework.Results: In total, 266 nodules were identified. For doctors without AI assistance, the FP was 0.617-0.650/scan and the sensitivity was 59.2-67.0%. For doctors with AI assistance, the FP was 0.067 to 0.2/scan and the sensitivity was 59.2-77.3% This AI-assisted program significantly reduced FP. The error-prone characteristics of lung nodules were central locations, ground-glass appearances, and small sizes. The AI-assisted program improved the detection of error-prone nodules.Conclusions: Detection of lung nodules is important for lung cancer treatment. When facing a large number of CT scans, error-prone nodules are a great challenge for doctors. The AI-assisted program improved the performance of detecting lung nodules, especially for error-prone nodules.     

不同分辨率单色液晶显示器对肺结节识读影响的初步研究

目的 评价不同分辨率的单色液晶显示器对肺结节检出效能的影响.方法 从数据库中在线选取胸部数字化放射成像(DR)影像图93幅:确诊图38幅、疑诊图32幅、正常图23幅(均由CT证实).将阳性病例按结节直径大小分为A、B两组,高、中、低年资医师各3名在3种不同分辨率的显示器上集中进行3次独立读图,对结节有无的评判采用5等分法:肯定有、可能有、不确定、可能无、肯定无,每名医师针对特定显示器上的每幅图像给出自己的信任等级.采用SPSS 13.0对结果进行统计分析.结果 高年资医师使用2 MP、3 MP、5 MP显示器识读A组结节时受试者操作特性(ROC)曲线下面积分别为0.643、0.686、0.739;中年资为0.636、0.682、0.717;低年资为0.623、0.656、0.721.识读B组结节时高年资医师为0.813、0.832、0.846;中年资为0.773、0.824、0.838;低年资为0.763、0.773、0.833.不同放射系统诊断效能比较差异无统计学意义(P＞0.05).结论 在不限制图像后处理工具的情况下,不同年资的医师在不同分辨率的显示器上识读A、B两组不同尺寸结节时诊断效能差异无统计学意义.     

Segmentation of pulmonary nodules in three-dimensional CT images by use of a spiral-scanning technique

Accurate segmentation of pulmonary nodules in computed tomography (CT) is an important and difficult task for computer-aided diagnosis of lung cancer. Therefore, the authors developed a novel automated method for accurate segmentation of nodules in three-dimensional (3D) CT. First, a volume of interest (VOI) was determined at the location of a nodule. To simplify nodule segmentation, the 3D VOI was transformed into a two-dimensional (2D) image by use of a key "spiral-scanning" technique, in which a number of radial lines originating from the center of the VOI spirally scanned the VOI from the "north pole" to the "south pole." The voxels scanned by the radial lines provided a transformed 2D image. Because the surface of a nodule in the 3D image became a curve in the transformed 2D image, the spiral-scanning technique considerably simplified the segmentation method and enabled reliable segmentation results to be obtained. A dynamic programming technique was employed to delineate the "optimal" outline of a nodule in the 2D image, which corresponded to the surface of the nodule in the 3D image. The optimal outline was then transformed back into 3D image space to provide the surface of the nodule. An overlap between nodule regions provided by computer and by the radiologists was employed as a performance metric for evaluating the segmentation method. The database included two Lung Imaging Database Consortium (LIDC) data sets that contained 23 and 86 CT scans, respectively, with 23 and 73 nodules that were 3 mm or larger in diameter. For the two data sets, six and four radiologists manually delineated the outlines of the nodules as reference standards in a performance evaluation for nodule segmentation. The segmentation method was trained on the first and was tested on the second LIDC data sets. The mean overlap values were 66% and 64% for the nodules in the first and second LIDC data sets, respectively, which represented a higher performance level than those of two existing segmentation methods that were also evaluated by use of the LIDC data sets. The segmentation method provided relatively reliable results for pulmonary nodule segmentation and would be useful for lung cancer quantification, detection, and diagnosis.     

Selective enhancement filters for nodules,vessels, and airway walls in two- and three-dimensional CT scans

Computer-aided diagnostic (CAD) schemes have been developed to assist radiologists in the early detection of lung cancer in radiographs and computed tomography (CT) images. In order to improve sensitivity for nodule detection, many researchers have employed a filter as a preprocessing step for enhancement of nodules. However, these filters enhance not only nodules, but also other anatomic structures such as ribs, blood vessels, and airway walls. Therefore, nodules are often detected together with a large number of false positives caused by these normal anatomic structures. In this study, we developed three selective enhancement filters for dot, line, and plane which can simultaneously enhance objects of a specific shape (for example, dot-like nodules) and suppress objects of other shapes (for example, line-like vessels). Therefore, as preprocessing steps, these filters would be useful for improving the sensitivity of nodule detection and for reducing the number of false positives. We applied our enhancement filters to synthesized images to demonstrate that they can selectively enhance a specific shape and suppress other shapes. We also applied our enhancement filters to real two-dimensional (2D) and three-dimensional (3D) CT images to show their effectiveness in the enhancement of specific objects in real medical images. We believe that the three enhancement filters developed in this study would be useful in the computerized detection of cancer in 2D and 3D medical images.     

ROC study of the effect of stereoscopic imaging on assessment of breast lesions

An observer performance study was conducted to evaluate the usefulness of assessing breast lesion characteristics with stereomammography. Stereoscopic image pairs of 158 breast biopsy tissue specimens were acquired with a GE Senographe 2000D full field digital mammography system using a 1.8x magnification geometry. A phantom-shift method equivalent to a stereo shift angle of +/- 3 degrees relative to a central axis perpendicular to the detector was used. For each specimen, two pairs of stereo images were taken at approximately orthogonal orientations. The specimens contained either a mass, microcalcifications, both, or normal tissue. Based on pathological analysis, 39.9% of the specimens were found to contain malignancy. The digital specimen radiographs were displayed on a high resolution MegaScan CRT monitor driven by a DOME stereo display board using in-house developed software. Five MQSA radiologists participated as observers. Each observer read the 316 specimen stereo image pairs in a randomized order. For each case, the observer first read the monoscopic image and entered his/her confidence ratings on the presence of microcalcifications and/or masses, margin status, BI-RADS assessment, and the likelihood of malignancy. The corresponding stereoscopic images were then displayed on the same monitor and were viewed through stereoscopic LCD glasses. The observer was free to change the ratings in every category after stereoscopic reading. The ratings of the observers were analyzed by ROC methodology. For the 5 MQSA radiologists, the average Az value for estimation of the likelihood of malignancy of the lesions improved from 0.70 for monoscopic reading to 0.72 (p=0.04) after stereoscopic reading, and the average Az value for the presence of microcalcifications improved from 0.95 to 0.96 (p=0.02). The Az value for the presence of masses improved from 0.80 to 0.82 after stereoscopic reading, but the difference fell short of statistical significance (p=0.08). The visual assessment of margin clearance was found to have very low correlation with microscopic analysis with or without stereoscopic reading. This study demonstrates the potential of using stereomammography to improve the detection and characterization of mammographic lesions.     

Automated lung nodule classification following automated nodule detection on CT: a serial approach

We have evaluated the performance of an automated classifier applied to the task of differentiating malignant and benign lung nodules in low-dose helical computed tomography (CT) scans acquired as part of a lung cancer screening program. The nodules classified in this manner were initially identified by our automated lung nodule detection method, so that the output of automated lung nodule detection was used as input to automated lung nodule classification. This study begins to narrow the distinction between the "detection task" and the "classification task." Automated lung nodule detection is based on two- and three-dimensional analyses of the CT image data. Gray-level-thresholding techniques are used to identify initial lung nodule candidates, for which morphological and gray-level features are computed. A rule-based approach is applied to reduce the number of nodule candidates that correspond to non-nodules, and the features of remaining candidates are merged through linear discriminant analysis to obtain final detection results. Automated lung nodule classification merges the features of the lung nodule candidates identified by the detection algorithm that correspond to actual nodules through another linear discriminant classifier to distinguish between malignant and benign nodules. The automated classification method was applied to the computerized detection results obtained from a database of 393 low-dose thoracic CT scans containing 470 confirmed lung nodules (69 malignant and 401 benign nodules). Receiver operating characteristic (ROC) analysis was used to evaluate the ability of the classifier to differentiate between nodule candidates that correspond to malignant nodules and nodule candidates that correspond to benign lesions. The area under the ROC curve for this classification task attained a value of 0.79 during a leave-one-out evaluation.     

Computer-aided diagnostic scheme for lung nodule detection in digital chest radiographs by use of a multiple-template matching technique.

We have been developing a computer-aided diagnostic (CAD) scheme to assist radiologists in improving the detection of pulmonary nodules in chest radiographs, because radiologists can miss as many as 30% of pulmonary nodules in routine clinical practice. A key to the successful clinical application of a CAD scheme is to ensure that there are only a small number of false positives that are incorrectly reported as nodules by the scheme. In order to significantly reduce the number of false positives in our CAD scheme, we developed, in this study, a multiple-template matching technique, in which a test candidate can be identified as a false positive and thus eliminated, if its largest cross-correlation value with non-nodule templates is larger than that with nodule templates. We describe the technique for determination of cross-correlation values for test candidates with nodule templates and non-nodule templates, the technique for creation of a large number of nodule templates and non-nodule templates, and the technique for removal of nodulelike non-nodule templates and non-nodulelike nodule templates, in order to achieve a good performance. In our study, a large number of false positives (44.3%) were removed with reduction of a very small number of true positives (2.3%) by use of the multiple-template matching technique. We believe that this technique can be used to significantly improve the performance of CAD schemes for lung nodule detection in chest radiographs.     

An automated CT based lung nodule detection scheme using geometric analysis of signed distance field

The authors present a new computerized scheme to automatically detect lung nodules depicted on computed tomography (CT) images. The procedure is performed in the signed distance field of the CT images. To obtain an accurate signed distance field, CT images are first interpolated linearly along the axial direction to form an isotropic data set. Then a lung segmentation strategy is applied to smooth the lung border aiming to include as many juxtapleural nodules as possible while minimizing over segmentations of the lung regions. Potential nodule regions are then detected by locating local maximas of signed distances in each subvolume with values and the sizes larger than the smallest nodule of interest in the three-dimensional space. Finally, all detected candidates are scored by computing the similarity distance of their medial axis-like shapes obtained through a progressive clustering strategy combined with a marching cube algorithm from a sphere based shape. A free-response receiver operating characteristics curve is computed to assess the scheme performance. A performance test on 52 low-dose CT screening examinations that depict 184 verified lung nodules showed that during the initial stage the scheme achieved an asymptotic maximum sensitivity of 95.1% (175/184) with an average of 1200 suspicious voxels per CT examination. The nine missed nodules included two small solid nodules (with a diameter < or =3.1 mm) and seven nonsolid nodules. The final performance level after the similarity scoring stage was an absolute sensitivity level, namely, including the nine missed during the initial stage, of 81.5% (150/184) with 6.5 false-positive identifications per CT examination. This preliminary study demonstrates the feasibility of applying a simple and robust geometric model using the signed distance field to identify suspicious lung nodules. In the authors' data set the sensitivity of this scheme is not affected by nodule size. In addition to potentially being a stand alone approach, the signed distance field based method can be easily implemented as an initial filtering step in other computer-aided detection schemes.     

非对称卷积核YOLO V2网络的CT影像肺结节检测

肺癌一直是严重威胁人类健康的疾病之一,肺结节作为早期肺癌的一个重要征象,在肺癌的早期诊断与治疗中具有重要的意义。传统的CT影像肺结节检测方法不仅步骤繁琐、处理速度慢,而且对于结节的检出率及定位精度都亟待提高。提出一种基于非对称卷积核YOLO V2网络的CT影像肺结节检测方法:首先将连续的CT序列叠加构造为伪彩色数据集,以增强病变和健康组织的差异;然后将含有非对称卷积核的inception V3模块引入到YOLO V2网络中,构造出一种适用于肺结节检测的深度网络,一方面利用YOLO V2网络在目标检测上的优势,另一方面通过inception V3模块在网络的宽度与深度上进行扩增,以提取更加丰富的特征;为进一步提高结节的定位精度,对损失函数的设计与计算方法也进行一定的改进。为验证所提检测模型的性能,从LIDC-IDRI数据集中选取1 010个病例的CT图像用于训练和测试,在大于3 mm的肺结节中,检测敏感度为94.25%,假阳性率为8.50%。实验表明,所提出的肺结节检测方法不仅可以简化肺部CT图像的处理过程,而且在结节检测率及定位精度方面均优于传统方法,可为肺结节检测提供一种新思路。     

Computer-aided diagnosis of pulmonary nodules on CT scans: segmentation and classification using 3D active contours

We are developing a computer-aided diagnosis (CAD) system to classify malignant and benign lung nodules found on CT scans. A fully automated system was designed to segment the nodule from its surrounding structured background in a local volume of interest (VOI) and to extract image features for classification. Image segmentation was performed with a three-dimensional (3D) active contour (AC) method. A data set of 96 lung nodules (44 malignant, 52 benign) from 58 patients was used in this study. The 3D AC model is based on two-dimensional AC with the addition of three new energy components to take advantage of 3D information: (1) 3D gradient, which guides the active contour to seek the object surface, (2) 3D curvature, which imposes a smoothness constraint in the z direction, and (3) mask energy, which penalizes contours that grow beyond the pleura or thoracic wall. The search for the best energy weights in the 3D AC model was guided by a simplex optimization method. Morphological and gray-level features were extracted from the segmented nodule. The rubber band straightening transform (RBST) was applied to the shell of voxels surrounding the nodule. Texture features based on run-length statistics were extracted from the RBST image. A linear discriminant analysis classifier with stepwise feature selection was designed using a second simplex optimization to select the most effective features. Leave-one-case-out resampling was used to train and test the CAD system. The system achieved a test area under the receiver operating characteristic curve (A(z)) of 0.83 +/- 0.04. Our preliminary results indicate that use of the 3D AC model and the 3D texture features surrounding the nodule is a promising approach to the segmentation and classification of lung nodules with CAD. The segmentation performance of the 3D AC model trained with our data set was evaluated with 23 nodules available in the Lung Image Database Consortium (LIDC). The lung nodule volumes segmented by the 3D AC model for best classification were generally larger than those outlined by the LIDC radiologists using visual judgment of nodule boundaries.     

3.0T磁共振背景抑制弥散加权成像不同图像后处理方法观察盆腔淋巴结的比较

目的 比较3.0T磁共振背景抑制弥散加权成像(DWIBS)不同图像后处理方法对盆腔淋巴结的观察效果.方法 对47例病理证实的子宫颈癌病例行常规MRI及DWI扫描.DWI进行4种不同方法的最大信号强度投影(MIP)重建,即冠状6mmMIP组、横断6mmMIP组、横断10mmMIP组、三维全景组.分别记录不同大小、不同部位所显示的淋巴结数,比较不同图像后处理方法的淋巴结显示情况.由2位放射诊断医师全盲随机观察所有图像,比较常规MRI和弥散加权成像淋巴结的观察效果.结果 (1)短径<5 mm及位于双侧腹股沟区的淋巴结,4种图像后处理方法与DWI(b=1000 s/mm~2)原始图像显示的淋巴结差异均有统计学意义.4种图像后处理方法中以横断6 mm MIP为最佳.(2)弥散加权成像结合T_2WI发现淋巴结的能力明显优于常规MRI[(7.1±3.4)个/例比(5.3±2.5)个/例,P<0.05].结论 对于不同的图像后处理方法,横断6 mm MIP重建对于不同大小、部位的盆腔淋巴结显示最佳;而对于小于5 mm及腹股沟区的淋巴结,4种图像后处理方法误差较大,需结合薄层DWI原始图像观察.3.0 T磁共振DWIBS分辨率高,能够更加清晰、直观地显示盆腔淋巴结,为临床治疗及预后评价提供更多信息.     

Classification of lung nodules in CT scans using three-dimensional deep convolutional neural networks with a checkpoint ensemble method

Background

Accurately detecting and examining lung nodules early is key in diagnosing lung cancers and thus one of the best ways to prevent lung cancer deaths. Radiologists spend countless hours detecting small spherical-shaped nodules in computed tomography (CT) images. In addition, even after detecting nodule candidates, a considerable amount of effort and time is required for them to determine whether they are real nodules. The aim of this paper is to introduce a high performance nodule classification method that uses three dimensional deep convolutional neural networks (DCNNs) and an ensemble method to distinguish nodules between non-nodules.

Methods

In this paper, we use a three dimensional deep convolutional neural network (3D DCNN) with shortcut connections and a 3D DCNN with dense connections for lung nodule classification. The shortcut connections and dense connections successfully alleviate the gradient vanishing problem by allowing the gradient to pass quickly and directly. Connections help deep structured networks to obtain general as well as distinctive features of lung nodules. Moreover, we increased the dimension of DCNNs from two to three to capture 3D features. Compared with shallow 3D CNNs used in previous studies, deep 3D CNNs more effectively capture the features of spherical-shaped nodules. In addition, we use an alternative ensemble method called the checkpoint ensemble method to boost performance.

Results

The performance of our nodule classification method is compared with that of the state-of-the-art methods which were used in the LUng Nodule Analysis 2016 Challenge. Our method achieves higher competition performance metric (CPM) scores than the state-of-the-art methods using deep learning. In the experimental setup ESB-ALL, the 3D DCNN with shortcut connections and the 3D DCNN with dense connections using the checkpoint ensemble method achieved the highest CPM score of 0.910.

Conclusion

The result demonstrates that our method of using a 3D DCNN with shortcut connections, a 3D DCNN with dense connections, and the checkpoint ensemble method is effective for capturing 3D features of nodules and distinguishing nodules between non-nodules.

     

Computer-Assisted Detection of Pulmonary Nodules: Preliminary Observations Using a Prototype System with Multidetector-Row CT Data Sets

The continued revolution in multidetector-row CT (MDCT) scanning increases the quality of lung imaging but at the cost of a greater burden of data for review and interpretation. This article discusses our preliminary experience with prototype software for lung nodule detection and characterization using MDCT data sets. We discuss the potential role of computer-assisted detection (CAD) as applied to the automatic detection of lung nodules. We also review the process of CAD, outline its potential results, and explore how it may fit into existing radiology practice. Finally, we discuss MDCT data-acquisition parameters and how they may affect the performance of CAD.     

Investigating the low-dose limits of multidetector CT in lung nodule surveillance

The purpose of this study was to evaluate the factors limiting nodule detection in thoracic computed tomography (CT) and to determine whether prior knowledge of nodule size and attenuation, available from a baseline CT study, influences the minimum radiation dose at which nodule surveillance CT scans can be performed while maintaining current levels of nodule detectability. Multiple nodules varying in attenuation (-509 to + 110 HU) and diameter (1.6 to 9.5 mm) were layered in random and ordered sequences within 2 lung cylinders made of Rando lung material and suspended within a custom-built CT phantom. Multiple CT scans were performed at varying kVp (120, 100, and 80), mA (200, 150, 100, 50, 20, and 10), and beam collimation (5, 2.5, and 1.25 mm) on a four-row multidetector scanner (Lightspeed, General Electric, Milwaukee, WI) using 0.8 s gantry rotation. The corresponding range of radiation dose over which images were acquired was 0.3-26.4 mGy. Nine observers independently performed three specific tasks, namely: (1) To detect a 3.2 mm nodule of 23 HU; (2) To detect 3.2 mm nodules of varying attenuation (-509 to -154 HU); and (3) To detect nodules varying in size (1.6-9 mm) and attenuation (-509 to 110 HU). A two-alternative forced-choice test was used in order to determine the limits of nodule detection in terms of the proportion of correct responses (Pcorr, related to the area under the ROC curve) as a summary metric of observer performance. The radiation dose levels for detection of 99% of nodules in each task were as follows: Task 1 (1 mGy); Task 2 (5 mGy); and Task 3 (7 mGy). The corresponding interobserver confidence limits were 1, 5, and 10 mGy for Tasks 1, 2, and 3, respectively. There was a fivefold increase in the radiation dose required for detection of lower-density nodules (Tasks 1 to 2). Absence of prior knowledge of the nodule size and density (Task 3) corresponds to a significant increase in the minimum required radiation dose. Significant image degradation and reduction in observer performance for all tasks occur at a dose of < or = 1 mGy. It is concluded that the size and attenuation of a nodule strongly influence the radiation dose required for confident evaluation with a minimum threshold value of 1-2 mGy (minimum dose CT). A prior knowledge of nodule size and attenuation is available from the baseline CT scan and is an important consideration in minimizing the radiation exposure required for nodule detection with surveillance CT.     

A Comparison of Lung Nodule Segmentation Algorithms: Methods and Results from a Multi-institutional Study

Tumor volume estimation, as well as accurate and reproducible borders segmentation in medical images, are important in the diagnosis, staging, and assessment of response to cancer therapy. The goal of this study was to demonstrate the feasibility of a multi-institutional effort to assess the repeatability and reproducibility of nodule borders and volume estimate bias of computerized segmentation algorithms in CT images of lung cancer, and to provide results from such a study. The dataset used for this evaluation consisted of 52 tumors in 41 CT volumes (40 patient datasets and 1 dataset containing scans of 12 phantom nodules of known volume) from five collections available in The Cancer Imaging Archive. Three academic institutions developing lung nodule segmentation algorithms submitted results for three repeat runs for each of the nodules. We compared the performance of lung nodule segmentation algorithms by assessing several measurements of spatial overlap and volume measurement. Nodule sizes varied from 29 μl to 66 ml and demonstrated a diversity of shapes. Agreement in spatial overlap of segmentations was significantly higher for multiple runs of the same algorithm than between segmentations generated by different algorithms (p?<?0.05) and was significantly higher on the phantom dataset compared to the other datasets (p?<?0.05). Algorithms differed significantly in the bias of the measured volumes of the phantom nodules (p?<?0.05) underscoring the need for assessing performance on clinical data in addition to phantoms. Algorithms that most accurately estimated nodule volumes were not the most repeatable, emphasizing the need to evaluate both their accuracy and precision. There were considerable differences between algorithms, especially in a subset of heterogeneous nodules, underscoring the recommendation that the same software be used at all time points in longitudinal studies.