Improved Jacobian Eigen-Analysis Scheme for Accelerating Learning in Feedforward Neural Networks

An important problem in the learning process when training feedforward artificial neural networks is the occurrence of temporary minima which considerably slows down learning convergence. In a series of previous works, we analyzed this problem by deriving a dynamical system model which is valid in the vicinity of temporary minima caused by redundancy of nodes in the hidden layer. We also demonstrated how to incorporate the characteristics of the dynamical model into a constrained optimization algorithm that allows prompt abandonment of temporary minima and acceleration of learning. In this work, we revisit the constrained optimization framework in order to develop a closed-form solution for the evolution of critical dynamical system model parameters during learning in the vicinity of temporary minima. We show that this formalism is equivalent to matrix perturbation theory which was discussed in a previous work, but that the closed-form solution presented in the present paper allows for a weight update rule which is linear to the number of the network’s weights. In terms of computational complexity, this is equivalent to that of the simple back-propagation weight update rule. Simulations demonstrate the computational efficiency and effectiveness of this approach in reducing the time spent in the vicinity of temporary minima as suggested by the analysis.