Hierarchical evolution of heterogeneous neural networks

This paper describes a hierarchical evolutionary technique developed to design and train feedforward neural networks with different activation functions on their hidden-layer neurons (heterogeneous neural networks). At the upper level, a genetic algorithm is used to determine the number of neurons in the hidden layer and the type of the activation function of those neurons. At the second level, neural nets compete against each other across generations so that the nets with the lowest test errors survive. Finally, on the third level, a co-evolutionary approach is used to train each of the created networks by adjusting both the weights of the hidden-layer neurons and the parameters for their activation functions.