On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay

Training neural networks with batch normalization and weight decay has become a common practice in recent years. In this work, we show that their combined use may result in a surprising periodic behavior of optimization dynamics: the training process regularly exhibits destabilizations that, however...

Ausführliche Beschreibung

Bibliographische Detailangaben

Veröffentlicht in:	arXiv.org 2022-01
Hauptverfasser:	Lobacheva, Ekaterina, Kodryan, Maxim, Chirkova, Nadezhda, Malinin, Andrey, Vetrov, Dmitry
Format:	Artikel
Sprache:	eng
Schlagworte:	Convergence Decay Neural networks Training Weight
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!