Continual Transformers: Redundancy-Free Attention for Online Inference
International Conference on Learning Representations, 2023 Transformers in their common form are inherently limited to operate on whole token sequences rather than on one token at a time. Consequently, their use during online inference on time-series data entails considerable redundancy due to the o...
Gespeichert in:
Hauptverfasser: | , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | International Conference on Learning Representations, 2023 Transformers in their common form are inherently limited to operate on whole
token sequences rather than on one token at a time. Consequently, their use
during online inference on time-series data entails considerable redundancy due
to the overlap in successive token sequences. In this work, we propose novel
formulations of the Scaled Dot-Product Attention, which enable Transformers to
perform efficient online token-by-token inference on a continual input stream.
Importantly, our modifications are purely to the order of computations, while
the outputs and learned weights are identical to those of the original
Transformer Encoder. We validate our Continual Transformer Encoder with
experiments on the THUMOS14, TVSeries and GTZAN datasets with remarkable
results: Our Continual one- and two-block architectures reduce the floating
point operations per prediction by up to 63x and 2.6x, respectively, while
retaining predictive performance. |
---|---|
DOI: | 10.48550/arxiv.2201.06268 |