CaDM: Codec-aware Diffusion Modeling for Neural-enhanced Video Streaming
Recent years have witnessed the dramatic growth of Internet video traffic, where the video bitstreams are often compressed and delivered in low quality to fit the streamer's uplink bandwidth. To alleviate the quality degradation, it comes the rise of Neural-enhanced Video Streaming (NVS), which...
Gespeichert in:
Hauptverfasser: | , , , , , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | Recent years have witnessed the dramatic growth of Internet video traffic,
where the video bitstreams are often compressed and delivered in low quality to
fit the streamer's uplink bandwidth. To alleviate the quality degradation, it
comes the rise of Neural-enhanced Video Streaming (NVS), which shows great
prospects for recovering low-quality videos by mostly deploying neural
super-resolution (SR) on the media server. Despite its benefit, we reveal that
current mainstream works with SR enhancement have not achieved the desired
rate-distortion trade-off between bitrate saving and quality restoration, due
to: (1) overemphasizing the enhancement on the decoder side while omitting the
co-design of encoder, (2) limited generative capacity to recover high-fidelity
perceptual details, and (3) optimizing the compression-and-restoration pipeline
from the resolution perspective solely, without considering color bit-depth.
Aiming at overcoming these limitations, we are the first to conduct an
encoder-decoder (i.e., codec) synergy by leveraging the inherent
visual-generative property of diffusion models. Specifically, we present the
Codec-aware Diffusion Modeling (CaDM), a novel NVS paradigm to significantly
reduce streaming delivery bitrates while holding pretty higher restoration
capacity over existing methods. First, CaDM improves the encoder's compression
efficiency by simultaneously reducing resolution and color bit-depth of video
frames. Second, CaDM empowers the decoder with high-quality enhancement by
making the denoising diffusion restoration aware of encoder's resolution-color
conditions. Evaluation on public cloud services with OpenMMLab benchmarks shows
that CaDM effectively saves up to 5.12 - 21.44 times bitrates based on common
video standards and achieves much better recovery quality (e.g., FID of 0.61)
over state-of-the-art neural-enhancing methods. |
---|---|
DOI: | 10.48550/arxiv.2211.08428 |