Rapid Switching and Multi-Adapter Fusion via Sparse High Rank Adapters

In this paper, we propose Sparse High Rank Adapters (SHiRA) that directly finetune 1-2% of the base model weights while leaving others unchanged, thus, resulting in a highly sparse adapter. This high sparsity incurs no inference overhead, enables rapid switching directly in the fused mode, and signi...

Ausführliche Beschreibung

Bibliographische Detailangaben

Veröffentlicht in:	arXiv.org 2024-07
Hauptverfasser:	Bhardwaj, Kartikeya, Nilesh Prasad Pandey, Priyadarshi, Sweta, Ganapathy, Viswanath, Esteves, Rafael, Kadambi, Shreya, Borse, Shubhankar, Whatmough, Paul, Garrepalli, Risheek, Mart Van Baalen, Teague, Harris, Nagel, Markus
Format:	Artikel
Sprache:	eng
Schlagworte:	Adapters Switching
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!