Wonderful Matrices: More Efficient and Effective Architecture for Language Modeling Tasks

We prove the availability of inner product form position encoding in the state space dual algorithm and study the effectiveness of different position embeddings in the hybrid quadratic causal self-attention and state space dual algorithms. We propose inner function attention with dynamic mask, which...

Ausführliche Beschreibung

Bibliographische Detailangaben

Hauptverfasser:	Shi, Jingze, Wu, Bingheng, He, Lu, Jiang, Luchang
Format:	Artikel
Sprache:	eng
Schlagworte:	Computer Science - Artificial Intelligence Computer Science - Learning
Online-Zugang:	Volltext bestellen
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!