Dynamics of multilayer networks in the vicinity of temporary minima

A dynamical system model is derived for a single-output, two-layer neural network, which learns according to the back-propagation algorithm. Particular emphasis is placed on the analysis of the occurrence of temporary minima. The Jacobian matrix of the system is derived, whose eigenvalues characterize the evolution of learning. Temporary minima correspond to critical points of the phase plane trajectories, and the bifurcation of the Jacobian matrix eigenvalues signifies their abandonment. Following this analysis, we show that the employment of constrained optimization methods can decrease the time spent in the vicinity of this type of minima. A number of numerical results illustrates the analytical conclusions.