Safe Pretraining of Deep Language Models in a Synthetic Pseudo-Language

Absract This paper compares the pretraining of a transformer on natural language texts and on sentences of a synthetic pseudo-language. The artificial texts are automatically generated according to the rules written in a context-free grammar. The results of fine-tuning to complete tasks of the Russi...

Ausführliche Beschreibung

Bibliographische Detailangaben

Veröffentlicht in:	Doklady. Mathematics 2023-12, Vol.108 (Suppl 2), p.S494-S502
Hauptverfasser:	Gorbacheva, T. E., Bondarenko, I. Y.
Format:	Artikel
Sprache:	eng
Schlagworte:	Context Mathematics Mathematics and Statistics Texts
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!