Gaze and awareness prediction using neural network models

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for predicting gaze and awareness using neural network models. One of the methods includes obtaining sensor data (i) captured by one or more sensors of an autonomous vehicle and (ii) characterizing an agent in the vicinity of the autonomous vehicle in an environment at a current point in time. The sensor data is processed using a gaze prediction neural network to generate a gaze prediction that predicts a gaze of the agent at the current point in time. The gaze prediction neural network includes an embedding sub-network configured to process the sensor data to generate an embedding characterizing the agent, and a gaze sub-network configured to process the embedding to generate a gaze prediction. 用于使用神经网络模型预测凝视和意识的方法、系统和装置，包括编码在计算机存储介质上的计算机程序。其中一种方法包括获得(i)由自主交通工具的一个或多个传感器捕获并且(ii)表征在当前时间点处于环境中自主交通工具附近的代理的传感器数据。使用凝视预测神经网络来处理传感器数据，以生成预测代理在当前时间点的凝视的凝视预测。凝视预测神经网络包括嵌入子网络和凝视子网络，嵌入子网络被配置为处理传感器数据以生成表征代理的嵌入，凝视子网络被配置为处理嵌入以生