Survey of Research on Extended Models of Pre-trained Language Models

In recent years,the proposal of Transformer neural network has greatly promoted the development of pre-training technology.At present,pre-training models based on deep learning have become a research hotspot in the field of natural language processing.Since the end of 2018,BERT has achieved optimal results in multiple natural language processing tasks.A series of improved pre-training models based on BERT have been proposed one after another,and pre-training model extension models designed for various scenarios have also appeared.The expansion of pre-training models from single-language to tasks such as crosslanguage,multi-modality,and light-weighting has enabled natural language processing to enter a new era of pre-training.This paper mainly summarizes the research methods and research conclusions of lightweight pre-training models,knowledge-incorporated pre-training models,cross-modal pre-training language models and cross-language pre-training language models,as well as the main challenges faced by the pre