Monte Carlo Tree Search with macro-actions and heuristic route planning for the Multiobjective Physical Travelling Salesman Problem

This paper describes our entry to the Multiobjective Physical Travelling Salesman Problem (MO-PTSP) competition at the IEEE CIG 2013 conference. MO-PTSP combines the classical Travelling Salesman Problem with the task of steering a simulated spaceship on the 2-D plane, requiring that the controller minimises the three objectives of time taken, fuel consumed and damage incurred. Our entry to the MO-PTSP competition builds upon our winning entry to the previous (single-objective) PTSP competitions. This controller consists of two key components: a pre-planning stage using a classical TSP solver with a path cost measure that takes the physics of the problem into account, and a steering controller using Monte Carlo Tree Search (MCTS) with macro-actions (repeated actions), depth limiting and a heuristic fitness function for nonterminal states. We demonstrate that by modifying the two fitness functions we can produce effective behaviour in MO-PTSP without the need for major modifications to the overall architecture. The fitness functions used by our controller have several parameters, which must be set to ensure the best performance. Given the number of parameters and the difficulty of optimising a controller to satisfy multiple objectives in a search space which is many orders of magnitude larger than that encountered in a turn-based game such as Go, we show that informed hand tuning of parameters is insufficient for this task. We present an automatic parameter tuning method using the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) algorithm, which produced parameter settings that dominate our hand tuned parameters. Additionally we show that the robustness of the controller using hand tuned parameters can be improved by detecting when the controller is trapped in a poor quality local optimum and escaping by switching to an alternate fitness function.