Objective To study the application of hepatamethine cyanine near-infrared fluorescence (NIRF) dye IR-783 in the mouse models of human liver cancer exenografts, and to analyze the molecular mechanisms of the NIRF dye targeting tumor cells. Methods Luciferase-tagged HepG2 cells were inoculated subcutaneously into the nude mice. We detected the correlation of NIRF intensity and bioluminescence intensity (BIL) in the tumor region. Patient-derived xenograft (PDX) model was established in mouse by subrenal capsular implantation of clinic liver cancer specimen. After injecting the IR-783 dye, the interface between mouse kidney and the xenograft tumors was confirmed by NIRF analysis, and the tumor tissue in kidney was observed by pathology using H&E staining. The expression of CEA, AFP, HIF1α and OATP3A1 in the liver cancer tissue was detected by immunohistochemical staining. The intracellular retention of NIRF dyes was observed under fluorescence microscope after adding Mito Tracker or Lyso Tracker into cultured HepG2 cells. We added IR-783 in a co-culture system of HCCs and normal liver cells to test the specifical identification ability of IR-783 of the liver cancer cells. Results There was a good correlation between NIRF intensity and BIL intensity of the subcutaneous liver cancer xenograft region in nude mice. The margin between the mouse kidney tissue and xenograft tumors was clearly identified by IR-783.Compared with normal kidney tissue, CEA, HIF1α, OATP3A1 and AFP were highly expressed in the tumor region detected by IHC staining. The NIRF dye IR-783 was mainly accumulated in the mitochondria and lysosomes of cancer cells. GFP-tagged HepG2 cells could be recognized directly, whereas red fluorescence was not detected in normal liver cells. Conclusions IR-783 is a novel near-infrared fluorescent dye with tumor targeting and imaging properties. Its targeting ability may be related to the high expression of HIF1α and OATP3A1 in the liver cancer tissue.