Causal Interpretation of Self-Attention in Pre-Trained Transformers

We propose a causal interpretation of self-attention in the Transformer neural network architecture. We interpret self-attention as a mechanism that estimates a structural equation model for a given input sequence of symbols (tokens). The structural equation model can be interpreted, in turn, as a c...

Ausführliche Beschreibung

Bibliographische Detailangaben

Veröffentlicht in:	arXiv.org 2023-10
Hauptverfasser:	Rohekar, Raanan Y, Gurwicz, Yaniv, Shami Nisimov
Format:	Artikel
Sprache:	eng
Schlagworte:	Algorithms Neural networks Symbols
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!