‘Plain backpropagation’ and advanced optimization algorithms: A comparative study

We present a comparison of ‘plain backpropagation’ to other standard optimization algorithms. These algorithms include direct search as well as first and second order methods. All algorithms share the property of easy accessability. We chose a feedforward architecture for a network to perform function approximation. The results show a clear advantage of higher order methods with respect to quality of solution as well as execution times.