Expert-Token Resonance: Redefining MoE Routing through Affinity-Driven Active Selection

Mixture-of-Experts (MoE) architectures have emerged as a paradigm-shifting approach for large language models (LLMs), offering unprecedented computational efficiency. However, these architectures grapple with challenges of token distribution imbalance and expert homogenization, impeding optimal sema...

Ausführliche Beschreibung

Bibliographische Detailangaben

Veröffentlicht in:	arXiv.org 2024-08
Hauptverfasser:	Li, Jing, Sun, Zhijie, Lin, Dachao, He, Xuan, Lin, Yi, Zheng, Binfan, Zeng, Li, Zhao, Rongqian, Chen, Xin
Format:	Artikel
Sprache:	eng
Schlagworte:	Affinity Homogenization Large language models Lower bounds
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!