Heuristic dynamic programming using echo state network as online trainable adaptive critic

SUMMARYThe present paper proposes an implementation of a relatively new recurrent neural network architecture—the echo state network (ESN)–within the frame of heuristic dynamic programming. The ESN is trained online to estimate the utility function and to adapt the control policy of an embodied agent. With the advantage of an easy training algorithm, the ESN architecture offers a simple way to calculate the derivatives required for adapting the controller. Experimental results are provided to validate the proposed learning approach. Copyright © 2012 John Wiley & Sons, Ltd.