Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Models

The "state" of State Space Models (SSMs) represents their memory, which fades exponentially over an unbounded span. By contrast, Attention-based models have "eidetic" (i.e., verbatim, or photographic) memory over a finite span (context size). Hybrid architectures combine State Sp...

Ausführliche Beschreibung

Bibliographische Detailangaben

Veröffentlicht in:	arXiv.org 2024-12
Hauptverfasser:	Nunez, Elvis, Zancato, Luca, Bowman, Benjamin, Golatkar, Aditya, Xia, Wei, Soatto, Stefano
Format:	Artikel
Sprache:	eng
Schlagworte:	Attention Context Natural language State space models
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!