The Nesterov-Spokoiny Acceleration Achieves Strict \(o(1/k^2)\) Convergence

A lower bound result of Nesterov states that for a smooth convex objective \(f \in \mathscr{F}_{L}^{\infty,1} (\mathbb{R}^n)\), an algorithm that satisfies \( \mathbf{x}_{k+1} \in \mathbf{x}_0 + \mathrm{Lin} \{ \nabla f (\mathbf{x}_0), \cdots , \nabla f (\mathbf{x}_k) \} \) \((k\ge 0)\) cannot converge faster than \(\Omega ( 1/k^2 ) \) when \(k\) is small. In this paper, we show that when \(k\) is large, this worst-case lower bound is a bit overly pessimistic. We introduce a variant of an accelerated gradient algorithm of Nesterov and Spokoiny. We call this algorithm the Nesterov-Spokoiny Acceleration (NSA). The NSA algorithm simultaneously satisfies the following properties. 1. The sequence \(\{ \mathbf{x}_k \}_{k \in \mathbb{N}}\) governed by NSA obeys \( \mathbf{x}_{k+1} \in \mathbf{x}_0 + \mathrm{Lin} \{ \nabla f (\mathbf{x}_0), \cdots , \nabla f (\mathbf{x}_k) \} \) \((k\ge 0)\), and 2. For a smooth convex objective \(f \in \mathscr{F}_{L}^{\infty,1} (\mathbb{R}^n) \), the sequence \(\{ \mathbf{x}_k \}_{k \in \mathbb{N}}\) governed by NSA satisfies \( \limsup\limits_{k \to \infty } k^2 ( f (\mathbf{x}_k ) - f^* ) = 0 \), where \(f^* > -\infty\) is the minimum of \(f\). To our knowledge, NSA is the first algorithm that simultaneously satisfies items 1 and 2.