On the Influence of Masking Policies in Intermediate Pre-training

Current NLP models are predominantly trained through a two-stage "pre-train then fine-tune" pipeline. Prior work has shown that inserting an intermediate pre-training stage, using heuristic masking policies for masked language modeling (MLM), can significantly improve final performance. Ho...

Ausführliche Beschreibung

Bibliographische Detailangaben

Hauptverfasser:	Ye, Qinyuan, Li, Belinda Z, Wang, Sinong, Bolte, Benjamin, Ma, Hao, Yih, Wen-tau, Ren, Xiang, Khabsa, Madian
Format:	Artikel
Sprache:	eng
Schlagworte:	Computer Science - Computation and Language Computer Science - Learning
Online-Zugang:	Volltext bestellen
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!