Study of cost functions in Three Term Backpropagation for classification problems

Three term backpropagation (BP) network as proposed by Zweiri in 2003 has outperformed standard two term backpropagation. However, further studies on three term backpropagation in 2007 indicated that this network only surpassed standard BP for small scale datasets but not for medium and large scale datasets. It has also been observed that by using mean square error (MSE) as a cost function in three term BP has some drawbacks, and these include incorrect saturation and tend to trap in local minima, resulting in slow convergence and poor performance. In this study, thorough experiments on implementing various cost functions are executed to probe the effectiveness of Three Term BP network. The cost functions under investigations include mean square error (MSE), Bernoulli function, Modified cost function and Improved cost function. The results reveal that MSE is not an ideal cost function to be used for three term BP. Hence, alternative cost functions need to be considered when using BP network for classification problems.