What inductive bias gives good neural network training performance?

There has been an increased interest in the use of prior knowledge for training neural networks. Given a set of training examples and an initial domain theory, a neural network is constructed that fits the training examples by preprogramming some of the weights. The initialized neural network is then trained using backpropagation to refine the knowledge. This paper proposes a heuristic for determining the strength of the inductive bias by making use of gradient information in weight space in the direction of the programmed weights. The network starts its search in weight space where the gradient is maximal thus speeding-up convergence. Tests on a benchmark problem from molecular biology demonstrate that our heuristic, on average, reduces the training time by 60% compared to a random choice of the strength of the inductive bias; this performance is within 20% of the training time that can be achieved with the optimal inductive bias. The difference in generalization performance is not statistically significant.