Systems and methods for capturing image data for recreation in a virtual environment

In one embodiment, a method includes a computing system which may receive captured image data from a camera assembly. The camera assembly may include one or more cameras and an actuator to move the cameras. The image data may include frames that were captured while the actuator was moving the cameras. The computing system may determine for each frame, the time and position of the camera at the point that the image data was captured. The computing system may then generate an index of the captured image data and the associated time and position data. The index may be utilized for reconstructing the scene around the camera assembly from a viewpoint which is different from the viewpoint of any one of the cameras. This reconstruction may be performed based on one or more frames of the captured image data queried using the index, the viewpoint, and a time of interest.