Goal-Conditioned Predictive Coding for Offline Reinforcement Learning
Recent work has demonstrated the effectiveness of formulating decision making as supervised learning on offline-collected trajectories. Powerful sequence models, such as GPT or BERT, are often employed to encode the trajectories. However, the benefits of performing sequence modeling on trajectory da...
Gespeichert in:
Hauptverfasser: | , , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | Recent work has demonstrated the effectiveness of formulating decision making
as supervised learning on offline-collected trajectories. Powerful sequence
models, such as GPT or BERT, are often employed to encode the trajectories.
However, the benefits of performing sequence modeling on trajectory data remain
unclear. In this work, we investigate whether sequence modeling has the ability
to condense trajectories into useful representations that enhance policy
learning. We adopt a two-stage framework that first leverages sequence models
to encode trajectory-level representations, and then learns a goal-conditioned
policy employing the encoded representations as its input. This formulation
allows us to consider many existing supervised offline RL methods as specific
instances of our framework. Within this framework, we introduce
Goal-Conditioned Predictive Coding (GCPC), a sequence modeling objective that
yields powerful trajectory representations and leads to performant policies.
Through extensive empirical evaluations on AntMaze, FrankaKitchen and
Locomotion environments, we observe that sequence modeling can have a
significant impact on challenging decision making tasks. Furthermore, we
demonstrate that GCPC learns a goal-conditioned latent representation encoding
the future trajectory, which enables competitive performance on all three
benchmarks. |
---|---|
DOI: | 10.48550/arxiv.2307.03406 |