Homogeneous Second-Order Descent Framework: A Fast Alternative to Newton-Type Methods
This paper proposes a homogeneous second-order descent framework (HSODF) for nonconvex and convex optimization based on the generalized homogeneous model (GHM). In comparison to the Newton steps, the GHM can be solved by extremal symmetric eigenvalue procedures and thus grant an advantage in ill-con...
Gespeichert in:
Hauptverfasser: | , , , , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | This paper proposes a homogeneous second-order descent framework (HSODF) for
nonconvex and convex optimization based on the generalized homogeneous model
(GHM). In comparison to the Newton steps, the GHM can be solved by extremal
symmetric eigenvalue procedures and thus grant an advantage in ill-conditioned
problems. Moreover, GHM extends the ordinary homogeneous model (OHM) (Zhang et
al. 2022) to allow adaptiveness in the construction of the aggregated matrix.
Consequently, HSODF is able to recover some well-known second-order methods,
such as trust-region methods and gradient regularized methods, while
maintaining comparable iteration complexity bounds. We also study two specific
realizations of HSODF. One is adaptive HSODM, which has a parameter-free
$O(\epsilon^{-3/2})$ global complexity bound for nonconvex second-order
Lipschitz continuous objective functions. The other one is homotopy HSODM,
which is proven to have a global linear rate of convergence without strong
convexity. The efficiency of our approach to ill-conditioned and
high-dimensional problems is justified by some preliminary numerical results. |
---|---|
DOI: | 10.48550/arxiv.2306.17516 |