预训练词嵌入技术的演化与算法

目的:探究预训练词嵌入技术的重要性和主要演化路径,剖析其路径上的主流算法.方法:利用知识图谱分析预训练语言模型的基础知识及预训练词嵌入技术演化的关键路径,从算法角度分析其路径上代表性模型的内涵和优缺点.结果:词嵌入技术作为预训练语言模型的主要知识基础之一,包含Word2Vec、LSTM、Bi-LSTM和BERT等研究热...

Evolution and algorithm of pre-trained word embedding technology

Objective To study the importance and major evolution paths of pre-trained word embedding technology and analyze the mainstream algorithm on evolution paths of pre-trained word embedding technology. Methods The basic knowledge of pre-trained language embedding technology and the key paths of pre-trained word embedding technology Were analyzed using knowledge graphs. The connotation, advantages and disadvantages of representative model on the paths of pre-trained word embedding technology were analyzed by algorithm analysis. Results As one of the basic knowledge elements of pre-trained language embedding technology, word embedding technology includes the characteristics of studies on word2Vec, LSTM, Bi-LSTM and BERT. LSTM-based characteristics extraction technique and Transformer-based characteristics extraction technique are an important evolution path of word embedding technology. Model structure and algorithm analysis showed that the representation capability of model on the paths of pre-trained word embedding technology is becoming stronger and stronger. Conclusion Pre-trained language model-based word embedding technology can achieve word vector with more language context information and has become an important representation method of Chinese text in natural language processing field.