Training and/or utilizing machine learning models for use in natural language-based robot control

Techniques are disclosed that enable training of a target condition policy based on multiple data sets, where each data set describes a robotic task in a different manner. For example, the plurality of data sets can include: a target image data set in which a task is captured in a target image; a natural language instruction dataset in which the task is described in a natural language instruction; and a task ID data set, wherein the task is described by a task ID. In various embodiments, each of the plurality of data sets has a corresponding encoder, where the encoders are trained to generate a shared potential spatial representation of a corresponding task description. Additional or alternative techniques are disclosed that enable the use of a target conditional policy network to control a robot. For example, the robot can be controlled using the target conditional policy network based on free-form natural language input describing robot task (s). 公开了能够基于多个数据集来训练目标条件策略的技术，其中，每个数据集以不同方式描述机器人任务。例如，所述多个数据集能够包括：