Gradient Descent Algorithm For Near-Zero Learning Rate

A system and method for iteratively updating a parameter according to a gradient descent algorithm. In a given nth iteration of the method, one or more processors may determine a gradient value of a gradient vector of the parameter in a first dimension, determine a product value based at least in part on a sum of (i) the product value determined in an n−1th iteration and (ii) a product of the determined gradient value and a learning rate of the gradient descent algorithm, determine an updated parameter value according to a function including the product value, and update the parameter to equal the updated parameter value.