Multi Resolution Analysis (MRA) for Approximate Self-Attention

Transformers have emerged as a preferred model for many tasks in natural langugage processing and vision. Recent efforts on training and deploying Transformers more efficiently have identified many strategies to approximate the self-attention matrix, a key module in a Transformer architecture. Effec...

Ausführliche Beschreibung

Bibliographische Detailangaben

Veröffentlicht in:	arXiv.org 2022-07
Hauptverfasser:	Zeng, Zhanpeng, Pal, Sourav, Kline, Jeffery, Fung, Glenn M, Singh, Vikas
Format:	Artikel
Sprache:	eng
Schlagworte:	Empirical analysis Multiresolution analysis Wavelet analysis
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!