Multi-agent trajectory prediction

Methods and systems for training a trajectory prediction model and performing a vehicle maneuver include encoding a set of training data to generate encoded training vectors, where the training data includes trajectory information for agents over time. Trajectory scenarios are simulated based on the encoded training vectors, with each simulated trajectory scenario representing one or more agents with respective agent trajectories, to generate simulated training data. A predictive neural network model is trained using the simulated training data to generate predicted trajectory scenarios based on a detected scene.