Sparse Modular Activation for Efficient Sequence Modeling

Recent hybrid models combining Linear State Space Models (SSMs) with self-attention mechanisms have demonstrated impressive results across a range of sequence modeling tasks. However, current approaches apply attention modules statically and uniformly to all elements in the input sequences, leading...

Ausführliche Beschreibung

Bibliographische Detailangaben

Hauptverfasser:	Ren, Liliang, Liu, Yang, Wang, Shuohang, Xu, Yichong, Zhu, Chenguang, Zhai, ChengXiang
Format:	Artikel
Sprache:	eng
Schlagworte:	Computer Science - Computation and Language Computer Science - Learning
Online-Zugang:	Volltext bestellen
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!