A Mean Field Ansatz for Zero-Shot Weight Transfer

The pre-training cost of large language models (LLMs) is prohibitive. One cutting-edge approach to reduce the cost is zero-shot weight transfer, also known as model growth for some cases, which magically transfers the weights trained in a small model to a large model. However, there are still some t...

Ausführliche Beschreibung

Bibliographische Detailangaben

Veröffentlicht in:	arXiv.org 2024-08
Hauptverfasser:	Chen, Xingyuan, Kuang, Wenwei, Deng, Lei, Han, Wei, Bai, Bo, Goncalo dos Reis
Format:	Artikel
Sprache:	eng
Schlagworte:	Columnar structure Large language models Mean field theory Neural networks
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!