Generating scene flow labels for point clouds using object labels

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting scene flow. One of the methods includes obtaining a current point cloud representing an observed scene at a current time point; obtaining object label data that identifies a first three-dimensional region in the observed scene; determining, for each current three-dimensional point that is within the first three-dimensional region and using the object label data, a respective preceding position of the current three-dimensional point at a preceding time point in a reference frame of the sensor at the current time point; and generating, using the preceding positions, a scene flow label for the current point cloud that comprises a respective ground truth motion vector for each of a plurality of the current three-dimensional points.