Using simulations to identify differences between behaviors of manually-driven and autonomous vehicles

A simulation may be used to determine a difference between progress of a manually-driven vehicle and progress of a simulated autonomous vehicle. The method includes retrieving log data collected for the manually-driven vehicle driving along a route, generating a plurality of path segments for a portion of the route. The plurality of path segments corresponds to points in a lane that the manually-driven vehicle traveled through on the portion of the route. The method also includes running, using a software of the autonomous vehicle, a simulation of the autonomous vehicle driving along the plurality of path segments, extracting metrics from the log data and the simulation, and determining the difference between a first progress of the manually-driven vehicle and a second progress of the simulated autonomous vehicle based on the metrics.