Hebbian Learning from First Principles
Recently, the original storage prescription for the Hopfield model of neural networks -- as well as for its dense generalizations -- has been turned into a genuine Hebbian learning rule by postulating the expression of its Hamiltonian for both the supervised and unsupervised protocols. In these note...
Gespeichert in:
Hauptverfasser: | , , , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | Recently, the original storage prescription for the Hopfield model of neural
networks -- as well as for its dense generalizations -- has been turned into a
genuine Hebbian learning rule by postulating the expression of its Hamiltonian
for both the supervised and unsupervised protocols. In these notes, first, we
obtain these explicit expressions by relying upon maximum entropy extremization
\`a la Jaynes. Beyond providing a formal derivation of these recipes for
Hebbian learning, this construction also highlights how Lagrangian constraints
within entropy extremization force network's outcomes on neural correlations:
these try to mimic the empirical counterparts hidden in the datasets provided
to the network for its training and, the denser the network, the longer the
correlations that it is able to capture. Next, we prove that, in the big data
limit, whatever the presence of a teacher (or its lacking), not only these
Hebbian learning rules converge to the original storage prescription of the
Hopfield model but also their related free energies (and, thus, the statistical
mechanical picture provided by Amit, Gutfreund and Sompolinsky is fully
recovered). As a sideline, we show mathematical equivalence among standard Cost
functions (Hamiltonian), preferred in Statistical Mechanical jargon, and
quadratic Loss Functions, preferred in Machine Learning terminology. Remarks on
the exponential Hopfield model (as the limit of dense networks with diverging
density) and semi-supervised protocols are also provided. |
---|---|
DOI: | 10.48550/arxiv.2401.07110 |