Real-time photo-realistic visualization of 3D environments for enhanced tele-operation of vehicles

This paper describes a method for creating photorealistic three-dimensional (3D) models of real-world environments in real-time for the purpose of improving and extending the capabilities of vehicle tele-operation. Our approach utilizes the combined data from a laser scanner (for modeling 3D geometry) and a video camera (for modeling surface appearance). The sensors are mounted on a moving vehicle platform, and a photo-realistic 3D model of the vehicle's environment is generated and displayed to the remote operator in real time. Our model consists of three main components: a textured ground surface, textured or colorized non-ground objects, and a textured background for representing regions beyond the laser scanner's sensing horizon. Our approach enables many unique capabilities for vehicle tele-operation, including viewing the scene from virtual viewpoints (e.g., behind the vehicle or top down), seamless augmentation of the environment with digital objects, and improved robustness to transmission latencies and data dropouts.