Improving Stochastic Cubic Newton with Momentum
We study stochastic second-order methods for solving general non-convex optimization problems. We propose using a special version of momentum to stabilize the stochastic gradient and Hessian estimates in Newton's method. We show that momentum provably improves the variance of stochastic estimat...
Gespeichert in:
Hauptverfasser: | , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | We study stochastic second-order methods for solving general non-convex
optimization problems. We propose using a special version of momentum to
stabilize the stochastic gradient and Hessian estimates in Newton's method. We
show that momentum provably improves the variance of stochastic estimates and
allows the method to converge for any noise level. Using the cubic
regularization technique, we prove a global convergence rate for our method on
general non-convex problems to a second-order stationary point, even when using
only a single stochastic data sample per iteration. This starkly contrasts with
all existing stochastic second-order methods for non-convex problems, which
typically require large batches. Therefore, we are the first to demonstrate
global convergence for batches of arbitrary size in the non-convex case for the
Stochastic Cubic Newton. Additionally, we show improved speed on convex
stochastic problems for our regularized Newton methods with momentum. |
---|---|
DOI: | 10.48550/arxiv.2410.19644 |