H2O-Danube3 Technical Report

We present H2O-Danube3, a series of small language models consisting of H2O-Danube3-4B, trained on 6T tokens and H2O-Danube3-500M, trained on 4T tokens. Our models are pre-trained on high quality Web data consisting of primarily English tokens in three stages with different data mixes before final s...

Ausführliche Beschreibung

Bibliographische Detailangaben

Hauptverfasser:	Pfeiffer, Pascal, Singer, Philipp, Babakhin, Yauhen, Fodor, Gabor, Dhankhar, Nischay, Ambati, Sri Satish
Format:	Artikel
Sprache:	eng
Schlagworte:	Computer Science - Computation and Language Computer Science - Learning
Online-Zugang:	Volltext bestellen
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!