User Navigation Modeling, Rate-Distortion Analysis, and End-to-End Optimization for Viewport-Driven 360 ^\circ Video Streaming

The emerging technologies of Virtual Reality (VR) and 360^\circ video introduce new challenges for state-of-the-art video communication systems. Enormous data volume and spatial user navigation are unique characteristics of 360^\circ videos that necessitate a space-time effective allocation of the available network streaming bandwidth over the 360^\circ video content to maximize the Quality of Experience (QoE) delivered to the user. Towards this objective, we investigate a framework for viewport-driven rate-distortion optimized 360^\circ video streaming that integrates the user view navigation patterns and the spatiotemporal rate-distortion characteristics of the 360^\circ video content to maximize the delivered user viewport video quality, for the given network/system resources. The framework comprises a methodology for assigning dynamic navigation likelihoods over the 360^\circ video spatiotemporal panorama, induced by the user navigation patterns, an analysis and characterization of the 360^\circ video panorama's spatiotemporal rate-distortion characteristics that leverage preprocessed spatial tilling of the content, and an optimization problem formulation and solution that capture and aim to maximize the delivered expected viewport video quality, given a user's navigation patterns, the 360^\circ video encoding/streaming decisions, and the available system/network resources. We formulate a Markov model to capture the navigation patterns of a user over the 360^\circ video panorama and simultaneously extend our actual navigation datasets by synthesizing additional realistic navigation data. Moreover, we investigate the impact of using two different tile sizes for equirectangular tiling of the 360^\circ video panorama. Our experimental results demonstrate the