Developing Value Networks for Game 2048 with Reinforcement Learning

The game 2048 is a stochastic single-player game and several computer players have been developed in not only research work but also student projects. Among them, the most successful approach is based on N-tuple networks trained by reinforcement learning methods. Though there have been several works on computer players with deep neural networks, their performance were not as good in most cases. In our previous work, we designed policy networks and applied supervised learning, which resulted in an average score of 215,802. In this study, we tackle the problem with value networks and reinforcement learning methods, since value networks are important to combine with game-tree search methods. We investigate the training methods in several aspects, including batches of training, use of symmetry, network structures, and use of game-specific tricks. We then conduct a training for 240 hours with the best configuration. With the best value network obtained, we achieved an average score of 228,100 with the greedy (1-ply search) play, and furthermore an average score of 406,927 by combining it with the 3-ply expectimax search.