Global motion estimation with iterative optimization-based independent univariate model for action recognition

•We deeply analyze the characteristics of global motions in action recognition scenario and develop a novel independent uni-variate model for global motion representation. It is a simplified version of perspective model. The number of its parameters is same as that of affine model. It makes a good trade-off between robustness and complexity which is more suitable for action recognition applications.•We propose an iterative optimization scheme for global motion estimation. The outlier points with local and global motion will be gradually discarded by an adaptive threshold during each iteration and the estimation process is implemented in a coarse-to-fine manner. Moreover, the local motion field is estimated based on mixed and estimated global motion fields through a spatio-temporal threshold based scheme.•We evaluate our proposed global and local motion estimation scheme on action recognition tasks. The separated local motion is adopted as input instead of original mixed motion field. Extensive experiments on multiple deep neural networks and action recognition datasets demonstrate the effectiveness of our proposed method. Motion information used in the existed video action recognition schemes is mixing of global motion(GM) and local motion(LM). In fact, GM & LM have their respective semantic concepts. Thus, it is promising to decouple GM and LM from the mixed motions. Numerous efforts have been made on the design of global motion models for video encoding, video dejittering, video denoising, and so on. Nevertheless, some of the models are too basic to cover the camera motions in action recognition while others are over-complicated. In this paper, we focus on the characteristic of the action recognition and propose a novel independent univariate GM model. It ignores camera rotation, which appears rarely in action recognition videos, and represents the GM in x and y direction respectively. Furthermore, GM is position invariant because it is from the universal camera motion. Pixels with global motions are subjected to the same parametric model and pixels with mixed motion can be seen as outliers. Motivated by this, we develop an iterative optimization scheme for GM estimation which removes the outlier points step by step and estimates global motions in a coarse-to-fine manner. Finally, the LM is estimated through a Spatio-temporal threshold-based method. Experimental results demonstrate that the proposed GM model makes a better trade-off between the model com