A Limitation of Gradient Descent Learning

Over decades, gradient descent has been applied to develop learning algorithm to train a neural network (NN). In this brief, a limitation of applying such algorithm to train an NN with persistent weight noise is revealed. Let V({\mathbf w}) be the performance measure of an ideal NN. V({\mathbf w}) is applied to develop the gradient descent learning (GDL). With weight noise, the desired performance measure (denoted as {\mathcal{ J}}({\mathbf w}) ) is E[V(\tilde {\mathbf w})|{\mathbf w}] , where \tilde {\mathbf w} is the noisy weight vector. Applying GDL to train an NN with weight noise, the actual learning objective is clearly not V({\mathbf w}) but another scalar function {\mathcal{ L}}({\mathbf w}) . For decades, there is a misconception that {\mathcal{ L}}({\mathbf w}) = {\mathcal{ J}}({\mathbf w}) , and hence, the actual model attained by the GDL is the desired model. However, we show that it might not: 1) with persistent additive weight noise, the actual model attained is the desired model as {\mathcal{ L}}({\mathbf w}) = {\mathcal{ J}}({\mathbf w}) ; and 2) with persistent multiplicative weight noise, the actual model attained is unlikely the desired model as {\mathcal{ L}}({\mathbf w}) \neq {\mathcal{ J}}({\mathbf w}) . Accordingly, the properties of the models attained as compared with the desired models are analyzed and the learning curves are sketched. Simulation results on 1) a simple regression problem and 2) the MNIST handwritten digit recognition are presented to support our claims.