Elastography ultrasound (EUS) imaging has shown its effectiveness for diagnosis of tumors by providing additional information about tissue stiffness to the conventional B-mode ultrasound (BUS). However, due to the lack of EUS devices and experienced sonologists, EUS is not widely used, especially in rural areas. It is still a challenging task to improve the performance of the single-modal BUS-based computer-aided diagnosis (CAD) for tumors. In this work, we propose a novel transfer learning (TL)–based deep neural network (DNN) algorithm, named CW-PM-DNN, for the BUS-based CAD by transferring diagnosis knowledge from EUS during model training. CW-PM-DNN integrates both the feature-level and classifier-level knowledge transfer into a unified framework. In the feature-level TL, a bichannel DNN is learned by the cross-weight-based multimodal DL (MDL-CW) algorithm to transfer informative features from EUS to BUS. In the classifier-level TL, a projective model (PM)–based classifier is then embedded to the pretrained bichannel DNN to implement the parameter transfer in the classifier model at the second stage. The back-propagation procedure is then applied to optimize the whole CW-PM-DNN to further improve its performance. Experimental results on two bimodal ultrasound tumor datasets demonstrate that the proposed CW-PM-DNN achieves the best classification accuracy, sensitivity, and specificity of 89.02 ± 1.54%, 88.37 ± 4.72%, and 89.63 ± 4.06%, respectively, for the breast ultrasound dataset, and the corresponding values of 80.57 ± 3.41%, 76.67 ± 3.85%, and 83.94 ± 3.95%, respectively, for the prostate ultrasound dataset. The proposed two-stage TL-based CW-PM-DNN algorithm outperforms all the compared algorithms. It is also proved that the performance of the BUS-based CAD can be significantly improved by transferring the knowledge of EUS. It suggests that CW-PM-DNN has the potential for more applications in the field of medical image–based CAD.
