Content-dependent Frequency Domain based RDO Mode Decision

In this paper, we propose a content-dependent frequency domain based RDO (rate distortion optimization) mode decision algorithm such that an optimal mode decision is achieved with significantly reduced computational complexity. Specifically, based on the fact that quantized coefficients mainly focus on the lower frequency domain and few of them are non-zero entries especially in the low bit rate case, a computational savings can be achieved by using simplified transform to obtain lower frequency coefficients, as well estimate rate and distortion in the frequency domain. Therefore calculation on transform, quantization, inverse quantization can be decreased, entropy coding can be omitted, and inverse transform can be decreased in case of using simplified transform or omitted in other cases. Primary result shows averagely more than 46.5% computation reduction in RDO time with neglectable quality degradation. And proposed algorithm can also be combined with other algorithm designed in spatial domain for further improvement