Human observer detection experiments with mammograms and power-law noise

We determined contrast thresholds for lesion detection as a function of lesion size in both mammograms and filtered noise backgrounds with the same average power spectrum, P(f)=B/f3. Experiments were done using hybrid images with digital images of tumors added to digitized normal backgrounds, displayed on a monochrome monitor. Four tumors were extracted from digitized specimen radiographs. The lesion sizes were varied by digital rescaling to cover the range from 0.5 to 16 mm. Amplitudes were varied to determine the value required for 92% correct detection in two-alternative forced-choice (2AFC) and 90% for search experiments. Three observers participated, two physicists and a radiologist. The 2AFC mammographic results demonstrated a novel contrast-detail (CD) diagram with threshold amplitudes that increased steadily (with slope of 0.3) with increasing size for lesions larger than 1 mm. The slopes for prewhitening model observers were about 0.4. Human efficiency relative to these models was as high as 90%. The CD diagram slopes for the 2AFC experiments with filtered noise were 0.44 for humans and 0.5 for models. Human efficiency relative to the ideal observer was about 40%. The difference in efficiencies for the two types of backgrounds indicates that breast structure cannot be considered to be pure random noise for 2AFC experiments. Instead, 2AFC human detection with mammographic backgrounds is limited by a combination of noise and deterministic masking effects. The search experiments also gave thresholds that increased with lesion size. However, there was no difference in human results for mammographic and filtered noise backgrounds, suggesting that breast structure can be considered to be pure random noise for this task. Our conclusion is that, in spite of the fact that mammographic backgrounds have nonstationary statistics, models based on statistical decision theory can still be applied successfully to estimate human performance.     

Estimation of the noisy component of anatomical backgrounds.

The knowledge of the relationship that links radiation dose and image quality is a prerequisite to any optimization of medical diagnostic radiology. Image quality depends, on the one hand, on the physical parameters such as contrast, resolution, and noise, and on the other hand, on characteristics of the observer that assesses the image. While the role of contrast and resolution is precisely defined and recognized, the influence of image noise is not yet fully understood. Its measurement is often based on imaging uniform test objects, even though real images contain anatomical backgrounds whose statistical nature is much different from test objects used to assess system noise. The goal of this study was to demonstrate the importance of variations in background anatomy by quantifying its effect on a series of detection tasks. Several types of mammographic backgrounds and signals were examined by psychophysical experiments in a two-alternative forced-choice detection task. According to hypotheses concerning the strategy used by the human observers, their signal to noise ratio was determined. This variable was also computed for a mathematical model based on the statistical decision theory. By comparing theoretical model and experimental results, the way that anatomical structure is perceived has been analyzed. Experiments showed that the observer's behavior was highly dependent upon both system noise and the anatomical background. The anatomy partly acts as a signal recognizable as such and partly as a pure noise that disturbs the detection process. This dual nature of the anatomy is quantified. It is shown that its effect varies according to its amplitude and the profile of the object being detected. The importance of the noisy part of the anatomy is, in some situations, much greater than the system noise. Hence, reducing the system noise by increasing the dose will not improve task performance. This observation indicates that the tradeoff between dose and image quality might be optimized by accepting a higher system noise. This could lead to a better resolution, more contrast, or less dose.     

Evaluation of internal noise methods for Hotelling observer models

The inclusion of internal noise in model observers is a common method to allow for quantitative comparisons between human and model observer performance in visual detection tasks. In this article, we studied two different strategies for inserting internal noise into Hotelling model observers. In the first strategy, internal noise was added to the output of individual channels: (a) Independent nonuniform channel noise, (b) independent uniform channel noise. In the second strategy, internal noise was added to the decision variable arising from the combination of channel responses. The standard deviation of the zero mean internal noise was either constant or proportional to: (a) the decision variable's standard deviation due to the external noise, (b) the decision variable's variance caused by the external noise, (c) the decision variable magnitude on a trial to trial basis. We tested three model observers: square window Hotelling observer (HO), channelized Hotelling observer (CHO), and Laguerre-Gauss Hotelling observer (LGHO) using a four alternative forced choice (4AFC) signal known exactly but variable task with a simulated signal embedded in real x-ray coronary angiogram backgrounds. The results showed that the internal noise method that led to the best prediction of human performance differed across the studied model observers. The CHO model best predicted human observer performance with the channel internal noise. The HO and LGHO best predicted human observer performance with the decision variable internal noise. The present results might guide researchers with the choice of methods to include internal noise into Hotelling model observers when evaluating and optimizing medical image quality.     

Does image quality matter? Impact of resolution and noise on mammographic task performance

The purpose of this study was to examine the effects of different resolution and noise levels on task performance in digital mammography. This study created an image set with images at three different resolution levels, corresponding to three digital display devices, and three different noise levels, with noise magnitudes similar to full clinical dose, half clinical dose, and quarter clinical dose. The images were read by five experienced breast imaging radiologists. The data were then analyzed to compute two accuracy statistics (overall classification accuracy and lesion detection accuracy) and performance at four diagnostic tasks (detection of microcalcifications, benign masses, malignant masses, and discrimination of benign and malignant masses). Human observer results showed decreasing display resolution had little effect on overall classification accuracy and individual diagnostic task performance, but increasing noise caused overall classification accuracy to decrease by a statistically significant 21% as the breast dose went to one quarter of its normal clinical value. The noise effects were most prominent for the tasks of microcalcification detection and mass discrimination. When the noise changed from full clinical dose to quarter clinical dose, the microcalcification detection performance fell from 89% to 67% and the mass discrimination performance decreased from 93% to 79%, while malignant mass detection performance remained relatively constant with values of 88% and 84%, respectively. As a secondary aim, the image set was also analyzed by two observer models to examine whether their performance was similar to humans. Observer models differed from human observers and each other in their sensitivity to resolution degradation and noise. The primary conclusions of this study suggest that quantum noise appears to be the dominant image quality factor in digital mammography, affecting radiologist performance much more profoundly than display resolution.     

Human efficiency for symmetry detection on 3D bumpy surfaces

Based on the ideal observer analysis, we investigated sampling properties of image information used by human visual system, for symmetrical pattern discrimination on 3D bumpy surface. There were three models of ideal observer (IO) to perform the task: 2D-IO using 2D projection image (i.e., retinal image), 2.5D-IO using image transformed to canonical view, and 3D-IO using recovered pattern image of 2D plane. We measured discrimination thresholds on the task for each IO model and subjects, and calculated human statistical efficiency relative to each ideal observer. The results indicated for the detection of a diagonal symmetry in the bumpy surface that human performance was similar to 3D-IO. This implies that human observers use the structure of the bumpy surface to detect the diagonal symmetry.     

Robust fluoroscopic respiratory gating for lung cancer radiotherapy without implanted fiducial markers

For gated lung cancer radiotherapy, it is difficult to generate accurate gating signals due to the large uncertainties when using external surrogates and the risk of pneumothorax when using implanted fiducial markers. We have previously investigated and demonstrated the feasibility of generating gating signals using the correlation scores between the reference template image and the fluoroscopic images acquired during the treatment. In this paper, we present an in-depth study, aiming at the improvement of robustness of the algorithm and its validation using multiple sets of patient data. Three different template generating and matching methods have been developed and evaluated: (1) single template method, (2) multiple template method, and (3) template clustering method. Using the fluoroscopic data acquired during patient setup before each fraction of treatment, reference templates are built that represent the tumour position and shape in the gating window, which is assumed to be at the end-of-exhale phase. For the single template method, all the setup images within the gating window are averaged to generate a composite template. For the multiple template method, each setup image in the gating window is considered as a reference template and used to generate an ensemble of correlation scores. All the scores are then combined to generate the gating signal. For the template clustering method, clustering (grouping of similar objects together) is performed to reduce the large number of reference templates into a few representative ones. Each of these methods has been evaluated against the reference gating signal as manually determined by a radiation oncologist. Five patient datasets were used for evaluation. In each case, gated treatments were simulated at both 35% and 50% duty cycles. False positive, negative and total error rates were computed. Experiments show that the single template method is sensitive to noise; the multiple template and clustering methods are more robust to noise due to the smoothing effect of aggregation of correlation scores; and the clustering method results in the best performance in terms of computational efficiency and accuracy.     

Information-theoretic CAD system in mammography: entropy-based indexing for computational efficiency and robust performance

We have previously presented a knowledge-based computer-assisted detection (KB-CADe) system for the detection of mammographic masses. The system is designed to compare a query mammographic region with mammographic templates of known ground truth. The templates are stored in an adaptive knowledge database. Image similarity is assessed with information theoretic measures (e.g., mutual information) derived directly from the image histograms. A previous study suggested that the diagnostic performance of the system steadily improves as the knowledge database is initially enriched with more templates. However, as the database increases in size, an exhaustive comparison of the query case with each stored template becomes computationally burdensome. Furthermore, blind storing of new templates may result in redundancies that do not necessarily improve diagnostic performance. To address these concerns we investigated an entropy-based indexing scheme for improving the speed of analysis and for satisfying database storage restrictions without compromising the overall diagnostic performance of our KB-CADe system. The indexing scheme was evaluated on two different datasets as (i) a search mechanism to sort through the knowledge database, and (ii) a selection mechanism to build a smaller, concise knowledge database that is easier to maintain but still effective. There were two important findings in the study. First, entropy-based indexing is an effective strategy to identify fast a subset of templates that are most relevant to a given query. Only this subset could be analyzed in more detail using mutual information for optimized decision making regarding the query. Second, a selective entropy-based deposit strategy may be preferable where only high entropy cases are maintained in the knowledge database. Overall, the proposed entropy-based indexing scheme was shown to reduce the computational cost of our KB-CADe system by 55% to 80% while maintaining the system's diagnostic performance.     

Impact of resolution and noise characteristics of digital radiographic detectors on the detectability of lung nodules

One of the unanswered questions in digital radiography is the connection between physical image quality metrics and clinical detection performance. In this paper, we examine the impact of two physical metrics, resolution and noise, on the detectability of nodules in a pulmonary background for specific digital radiographic detectors. A detection experiment was performed on a simulated image set using anatomical backgrounds from a high-quality lung radiograph and three different simulated nodule sizes (2-3.5 mm). The resolution and noise of the resulting images were modified using existing routines to simulate a selenium-based and a cesium iodide-based flat-panel detector at comparable exposures. A location-known-exactly (LKE) observer performance experiment was performed in which four experienced chest radiologists and three physicists specializing in chest radiology scored the images. The data from the observer experiment were analyzed by receiver operating characteristic (ROC) methodology. The detectability, as measured by the parameter Az, was higher for the selenium detector than the cesium iodide detector for all nodule sizes by an average of 8.5%. For one nodule size (2.75 mm), the difference between detectors was statistically significant (p < 0.01). The findings indicate that for the particular task studied, the superior resolution performance of the selenium-based detector provided better detectability of subtle lung nodules even though the images had greater noise than images obtained with the cesium iodide detector.     

Amorphous selenium flat panel detectors for digital mammography: validation of a NPWE model observer with CDMAM observer performance experiments

Model observers have been developed which incorporate a specific imaging task, system performance, and human observer characteristics and can potentially overcome some of the limitations in using detective quantum efficiency for optimization and comparison of detectors. In this paper, a modified nonprewhitening matched filter (NPWE) model observer was developed and validated to predict object detectability for an amorphous selenium (a-Se) direct flat-panel imager (FPI) where aliasing is severe. A preclinical a-Se digital mammography FPI with 85 microm pixel size was used in this investigation. Its physical imaging properties including modulation transfer function (MTF), noise power spectrum, and DQE were fully characterized. An observer performance study was conducted by imaging the CDMAM 3.4 contrast-detail phantom designed specifically for digital mammography and presenting these images to a panel of seven observers. X-ray attenuation and scatter due to the phantom were determined experimentally for use in development of the model observer. The observer study results were analyzed via threshold averaging and signal detection theory (SDT) based techniques to produce contrast-detail curves where threshold contrast is plotted as a function of disk diameter. Validity of the model was established using SDT analysis of the experimental data. The effect of aliasing on the detectability of small diameter disks was determined using the NPWE model observer. The signal spectrum was calculated using the presampling MTF of the detector with and without including the aliased terms. Our results indicate that the NPWE model based on Fourier domain parameters provides reasonable prediction of object detectability for the signal-known-exactly task in uniform image noise for a-Se direct FPI.     

Effect of dose reduction on the detection of mammographic lesions: a mathematical observer model analysis

The effect of reduction in dose levels normally used in mammographic screening procedures on the detection of breast lesions were analyzed. Four types of breast lesions were simulated and inserted into clinically-acquired digital mammograms. Dose reduction by 50% and 75% of the original clinically-relevant exposure levels were simulated by adding corresponding simulated noise into the original mammograms. The mammograms were converted into luminance values corresponding to those displayed on a clinical soft-copy display station and subsequently analyzed by Laguerre-Gauss and Gabor channelized Hotelling observer models for differences in detectability performance with reduction in radiation dose. Performance was measured under a signal known exactly but variable detection task paradigm in terms of receiver operating characteristics (ROC) curves and area under the ROC curves. The results suggested that luminance mapping of digital mammograms affects performance of model observers. Reduction in dose levels by 50% lowered the detectability of masses with borderline statistical significance. Dose reduction did not have a statistically significant effect on detection of microcalcifications. The model results indicate that there is room for optimization of dose level in mammographic screening procedures.     

Search for lesions in mammograms: statistical characterization of observer responses

We investigate human performance for visually detecting simulated microcalcifications and tumors embedded in x-ray mammograms as a function of signal contrast and the number of possible signal locations. Our results show that performance degradation with an increasing number of locations is well approximated by signal detection theory (SDT) with the usual Gaussian assumption. However, more stringent statistical analysis finds a departure from Gaussian assumptions for the detection of microcalcifications. We investigated whether these departures from the SDT Gaussian model could be accounted for by an increase in human internal response correlations arising from the image-pixel correlations present in 1/f spectrum backgrounds and/or observer internal response distributions that departed from the Gaussian assumption. Results were consistent with a departure from the Gaussian response distributions and suggested that the human observer internal responses were more compact than the Gaussian distribution. Finally, we conducted a free search experiment where the signal could appear anywhere within the image. Results show that human performance in a multiple-alternative forced-choice experiment can be used to predict performance in the clinically realistic free search experiment when the investigator takes into account the search area and the observers' inherent spatial imprecision to localize the targets.     

Disentangling signal from noise in visual contrast discrimination.

Human ability to detect stimulus changes (Delta C) decreases with increasing reference level (C). Because detection performance reflects the signal-to-noise ratio within the relevant sensory brain module, this behavior can be accounted for in two extreme ways: first, the internal response change Delta R evoked by a constant Delta C decreases with C (that is, the transducer R = f(C) displays a compressive nonlinearity), whereas the internal noise is independent of R; second, Delta R is constant with C but the noise level increases with R. A newly discovered constraint on human decision-making helps solve this century-old problem: in a detection task where multiple changes occur with equal probabilities, observers use a unique response criterion to decide whether a change has occurred. For contrast discrimination, our results supported the first account above: human performance was limited by the contrast transducer nonlinearity and an almost constant noise.     

A comparison of physical image quality indices and observer performance in the radiographic detection of nylon beads

Although various models have been proposed in an attempt to predict the usefulness of a radiographic image in terms of its physical characteristics, no previous work has shown whether a single physical image quality index, such as a signal-to-noise ratio, can reliably predict the performance of a human observer over a broad range of image characteristics. We studied the relationship between physical and visual image quality for the task of detecting nylon beads in radiographs. Thirty-seven imaging cases with different combinations of physical image characteristics were considered; these included variations in object size and magnification, X-ray beam quality, screen-film system, screen-film contact, film density and illumination, and viewing distance. For each imaging case, visual image quality was quantified in terms of observer performance in a 2AFC visual detection experiment. Physical image quality indices were calculated according to eight different models of the detection process; these indices combined data regarding object size and attenuation, screen-film system MTF, film gradient, noise Wiener spectrum, and visual system response. The results of this work indicate that, for the conditions studied, human observer detection performance most closely resembles that of a sub-optimal statistical decision process.     

Optimization of a tomosynthesis system for the detection of lung nodules

Mathematical observers that track human performance can be used to reduce the number of human observer studies needed to optimize imaging systems. The performance of human observers for the detection of a 3.6 mm lung nodule in anatomical backgrounds was measured as a function of varying tomosynthetic angle and compared with mathematical observers. The human observer results showed a dramatic increase in the percent of correct responses, from 80% in the projection images to 96% in the projection images with a tomosynthetic angle of just 3 degrees. This result suggests the potential usefulness of the scanned beam digital x-ray system for this application. Given the small number of images (40) used per tomosynthetic angle and the highly nonstationary statistical nature of the backgrounds, the nonprewhitening eye observer achieved a higher performance than the channelized Hotelling observer using a Laguerre-Gauss basis. The channelized Hotelling observer with internal noise and the eye filter matched to the projection data were shown to track human performance as the tomosynthetic angle changed. The validation of these mathematical observers extends their applicability to the optimization of tomosynthesis systems.     

Signal detectability in digital radiography: spatial domain figures of merit

The usefulness of Fourier-based measures of imaging performance has come into question for the evaluation of digital imaging systems. Figures of merit such as detective quantum efficiency (DQE) based on Fourier domain parameters are relevant for linear, shift-invariant systems with stationary noise. However, no digital imaging system is shift invariant, and realistic images do not satisfy the stationarity condition. Our methods for the task-based evaluation of imaging systems, based on signal detectability in the spatial domain, do not require such assumptions. We have computed the performance of ideal and quasi-ideal observers for the task of signal detection in digital radiography. Signal detectability in terms of an observer signal-to-noise-ratio (SNR) has been compared to results obtained from a Monte Carlo simulation of the digital image-acquisition process. The simulation incorporates the effects of random amplification and secondary quantum blur, integration over pixel area, and electronic noise. The observer figures of merit that have been previously shown to bracket human performance directly specify the usefulness of the images for the stated diagnostic task. In addition, the observer figures of merit give a task-dependent measure of imaging system efficiency in terms of the ratio of an output SNR2 to an input SNR2. Thus, the concept of "detective quantum efficiency" reappears in a natural way but based in the spatial domain and not dependent on shift invariance and stationarity assumptions. With respect to the optimum amount of system blur, our simulations indicate that under certain task-dependent conditions, large signals are fairly insensitive to blur in the x-ray transducer, while an optimum blur is found for small signals.     

Mutual information-based template matching scheme for detection of breast masses: from mammography to digital breast tomosynthesis

Development of a computational decision aid for a new medical imaging modality typically is a long and complicated process. It consists of collecting data in the form of images and annotations, development of image processing and pattern recognition algorithms for analysis of the new images and finally testing of the resulting system. Since new imaging modalities are developed more rapidly than ever before, any effort for decreasing the time and cost of this development process could result in maximizing the benefit of the new imaging modality to patients by making the computer aids quickly available to radiologists that interpret the images. In this paper, we make a step in this direction and investigate the possibility of translating the knowledge about the detection problem from one imaging modality to another. Specifically, we present a computer-aided detection (CAD) system for mammographic masses that uses a mutual information-based template matching scheme with intelligently selected templates. We presented principles of template matching with mutual information for mammography before. In this paper, we present an implementation of those principles in a complete computer-aided detection system. The proposed system, through an automatic optimization process, chooses the most useful templates (mammographic regions of interest) using a large database of previously collected and annotated mammograms. Through this process, the knowledge about the task of detecting masses in mammograms is incorporated in the system. Then, we evaluate whether our system developed for screen-film mammograms can be successfully applied not only to other mammograms but also to digital breast tomosynthesis (DBT) reconstructed slices without adding any DBT cases for training. Our rationale is that since mutual information is known to be a robust inter-modality image similarity measure, it has high potential of transferring knowledge between modalities in the context of the mass detection task. Experimental evaluation of the system on mammograms showed competitive performance compared to other mammography CAD systems recently published in the literature. When the system was applied “as-is” to DBT, its performance was notably worse than that for mammograms. However, with a simple additional preprocessing step, the performance of the system reached levels similar to that obtained for mammograms. In conclusion, the presented CAD system not only performed competitively on screen-film mammograms but it also performed robustly on DBT showing that direct transfer of knowledge across breast imaging modalities for mass detection is in fact possible.     

Ideal versus human observer for long-tailed point spread functions: does deconvolution help?

The ideal observer represents a Bayesian approach to performing detection tasks. Since such tasks are frequently used as a prototype tasks for radiological imaging systems, the detectability measured at the output of an ideal detector can be used as a figure of merit to characterize the imaging system. For the detectability achieved by the ideal observer to be a good figure of merit, it should predict the ability of the human observer to perform the same detection task. Of great general interest, especially to the medical community, are imaging devices with long-tailed point spread functions (PSFs). Such PSFs may occur due to septal penetration in collimators, veiling glare in image intensifiers or scattered radiation in the body. We have investigated the effect that this type of PSF has on human visual signal detection and whether any improvement in performance can be gained by deconvolving the tails of the PSF. For the ideal observer, it is straightforward to show that the performance is independent of any linear, invertible deconvolution filter. Our psychophysical studies show, however, that performance of the human observer is indeed improved by deconvolution. The ideal observer is, therefore, not a good predictor of human observer performance for detection of a signal imaged through a long-tailed PSF. We offer some explanations for this discrepancy by using some characteristics of the visual process and suggest a standard of comparison for the human observer that takes into account these characteristics. A look at the performance of the non-prewhitening (npw) ideal observer, before and after deconvolution, also brings some good insight into this study.     

Perceptual learning improves efficiency by re-tuning the decision 'template' for position discrimination

Visual position discrimination improves with practice; however, the mechanism(s) underlying this improvement are not yet known. We used positional noise to explore the underlying neural mechanisms and found that position discrimination improved with practice over a range of noise levels. This improvement can be largely explained by an increasing efficiency with which observers used positional information in the stimulus. In a second experiment, we tested the hypothesis that the improved efficiency reflects a re-tuning of the observers' perceptual 'template'--the weightings of inputs from basic visual mechanisms--to more closely match the ideal template required to perform the perceptual task. Using a new technique to measure which parts of the stimulus influenced the observer's performance, we were able to record the re-tuning of the decision template across training sessions; we found a robust and steady increase in template efficiency during learning.     

Suitability of new anode materials in mammography: dose and subject contrast considerations using Monte Carlo simulation

Mammography is the technique with the highest sensitivity and specificity, for the early detection of nonpalpable lesions associated with breast cancer. As screening mammography refers to asymptomatic women, the task of optimization between the image quality and the radiation dose is critical. A way toward optimization could be the introduction of new anode materials. A method for producing the x-ray spectra of different anode/filter combinations is proposed. The performance of several mammographic spectra, produced by both existing and theoretical anode materials, is evaluated, with respect to their dose and subject contrast characteristics, using a Monte Carlo simulation. The mammographic performance is evaluated utilizing a properly designed mathematical phantom with embedded inhomogeneities, irradiated with different spectra, based on combinations of conventional and new (Ru, Ag) anode materials, with several filters (Mo, Rh, Ru, Ag, Nb, Al). An earlier developed and validated Monte Carlo model, for deriving both image and dose characteristics in mammography, was utilized and overall performance results were derived in terms of subject contrast to dose ratio and squared subject contrast to dose ratio. Results demonstrate that soft spectra, mainly produced from Mo, Rh, and Ru anodes and filtered with k-edge filters, provide increased subject contrast for inhomogeneities of both small size, simulating microcalcifications and low density, simulating masses. The harder spectra (W and Ag anode) come short in the discrimination task but demonstrate improved performance when considering the dose delivered to the breast tissue. As far as the overall performance is concerned, new theoretical spectra demonstrate a noticeable good performance that is similar, and in some cases better compared to commonly used systems, stressing the possibility of introducing new materials in mammographic practice as a possible contribution to its optimization task. In the overall optimization task in terms of subject contrast to dose ratio, tube voltage was found to have a minor effect, while with respect to the filter material, a lesion specific performance was noticed, with Al filtered spectra showing improved characteristics in case of the inhomogeneities simulating microcalcifications, while softer k-edge filtered spectra are more suitable for the discrimination of inhomogeneities simulating masses.