In Search of "Better-than-DCT" Unitary Transforms for Encoding of Residual Signals

It is well known that the discrete cosine transform (DCT) closely approximates the optimal Karhunen-Loève transform (KLT) under the first-order stationary Markov condition with a strong inter-pixel correlation. However, if the inter-pixel correlation is weak or becomes negative, transforms other than the DCT would possibly become better. In this letter, we present a design framework for finding new unitary transforms according to two principles: 1) they are indeed more efficient than the DCT in encoding of signals with a weak or negative inter-pixel correlation (e.g., the residual signals after motion-compensation or intra- prediction) and 2) each has a fixed transform matrix (i.e., signal-independent) and can be implemented as efficiently as the DCT. Some of the new transforms will be used to demonstrate an improved R-D performance in the video coding scenario.