A Critical View of Vision-Based Long-Term Dynamics Prediction Under Environment Misalignment
Dynamics prediction, which is the problem of predicting future states of scene objects based on current and prior states, is drawing increasing attention as an instance of learning physics. To solve this problem, Region Proposal Convolutional Interaction Network (RPCIN), a vision-based model, was pr...
Gespeichert in:
Hauptverfasser: | , , , , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | Dynamics prediction, which is the problem of predicting future states of
scene objects based on current and prior states, is drawing increasing
attention as an instance of learning physics. To solve this problem, Region
Proposal Convolutional Interaction Network (RPCIN), a vision-based model, was
proposed and achieved state-of-the-art performance in long-term prediction.
RPCIN only takes raw images and simple object descriptions, such as the
bounding box and segmentation mask of each object, as input. However, despite
its success, the model's capability can be compromised under conditions of
environment misalignment. In this paper, we investigate two challenging
conditions for environment misalignment: Cross-Domain and Cross-Context by
proposing four datasets that are designed for these challenges: SimB-Border,
SimB-Split, BlenB-Border, and BlenB-Split. The datasets cover two domains and
two contexts. Using RPCIN as a probe, experiments conducted on the combinations
of the proposed datasets reveal potential weaknesses of the vision-based
long-term dynamics prediction model. Furthermore, we propose a promising
direction to mitigate the Cross-Domain challenge and provide concrete evidence
supporting such a direction, which provides dramatic alleviation of the
challenge on the proposed datasets. |
---|---|
DOI: | 10.48550/arxiv.2305.07648 |