Flash STU: Fast Spectral Transform Units

Flash STU: Fast Spectral Transform Units

This paper describes an efficient, open source PyTorch implementation of the Spectral Transform Unit. We investigate sequence prediction tasks over several modalities including language, robotics, and simulated dynamical systems. We find that for the same parameter count, the STU and its variants ou...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:arXiv.org 2024-09
Hauptverfasser: Liu, Y Isabel, Nguyen, Windsor, Devre, Yagiz, Dogariu, Evan, Majumdar, Anirudha, Hazan, Elad
Format: Artikel
Sprache:eng
Schlagworte:
Robotics
State space models
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:This paper describes an efficient, open source PyTorch implementation of the Spectral Transform Unit. We investigate sequence prediction tasks over several modalities including language, robotics, and simulated dynamical systems. We find that for the same parameter count, the STU and its variants outperform the Transformer as well as other leading state space models across various modalities.
ISSN:2331-8422