Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters

Exploiting activation sparsity is a promising approach to significantly accelerating the inference process of large language models (LLMs) without compromising performance. However, activation sparsity is determined by activation functions, and commonly used ones like SwiGLU and GeGLU exhibit limite...

Ausführliche Beschreibung

Bibliographische Detailangaben

Hauptverfasser:	Song, Yixin, Xie, Haotong, Zhang, Zhengyan, Wen, Bo, Ma, Li, Mi, Zeyu, Chen, Haibo
Format:	Artikel
Sprache:	eng
Schlagworte:	Computer Science - Computation and Language Computer Science - Learning
Online-Zugang:	Volltext bestellen
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!