SURFEL MAP EDITING

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing an initial representation of an environment to generate an updated representation that includes representations of one or more objects there were obscured in the initial representation. One of the methods includes obtaining initial surfel data comprising a plurality of surfels; determining a plurality of non-static surfels from the plurality of surfels; obtaining guidance data that characterizes a prediction for static surfaces in one or more first regions of the environment that were obscured by the plurality of non-static surfels in the initial surfel data; and processing the guidance data and the initial surfel data to generate final surfel data that comprises, for each first region of the environment, a plurality of predicted surfels that represent static surfaces in the first region that are partially or wholly obscured by objects represented by respective non-static surfels.