Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrations
Providing densely shaped reward functions for RL algorithms is often exceedingly challenging, motivating the development of RL algorithms that can learn from easier-to-specify sparse reward functions. This sparsity poses new exploration challenges. One common way to address this problem is using dem...
Gespeichert in:
Hauptverfasser: | , , , , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | Providing densely shaped reward functions for RL algorithms is often
exceedingly challenging, motivating the development of RL algorithms that can
learn from easier-to-specify sparse reward functions. This sparsity poses new
exploration challenges. One common way to address this problem is using
demonstrations to provide initial signal about regions of the state space with
high rewards. However, prior RL from demonstrations algorithms introduce
significant complexity and many hyperparameters, making them hard to implement
and tune. We introduce Monte Carlo Augmented Actor Critic (MCAC), a parameter
free modification to standard actor-critic algorithms which initializes the
replay buffer with demonstrations and computes a modified $Q$-value by taking
the maximum of the standard temporal distance (TD) target and a Monte Carlo
estimate of the reward-to-go. This encourages exploration in the neighborhood
of high-performing trajectories by encouraging high $Q$-values in corresponding
regions of the state space. Experiments across $5$ continuous control domains
suggest that MCAC can be used to significantly increase learning efficiency
across $6$ commonly used RL and RL-from-demonstrations algorithms. See
https://sites.google.com/view/mcac-rl for code and supplementary material. |
---|---|
DOI: | 10.48550/arxiv.2210.07432 |