Avatar animation using Markov decision process policies

Real-time animation of a virtual character can be provided by pre-calculating a control policy that identifies a suitable animation clip to animate the next movement of the virtual character. The control policy can be calculated using a Markov decision process (MDP) and can specify an action to take (e.g., a movement) when the virtual character is in a particular state (e.g., in a particular position or pose). The control policy can be determined based on a discounted set of rewards or punishments associated with various actions that can occur in each state. The control policies can be pre-calculated, offline from a runtime animation engine (e.g., in an augmented reality display). The runtime animation engine can use the control policies to select an animation clip to move the virtual character through an environment in real time.