Exploiting inter-frame regional correlation for efficient action recognition

•We proposed a novel temporal feature extraction method for action recognition.•The novel method explores inter-frame correlation on the regional level.•Our method achieves state-of-the-art performance on benchmark datasets. Temporal feature extraction is an important issue in video-based action recognition. Optical flow is a popular method to extract temporal feature, which produces excellent performance thanks to its capacity of capturing pixel-level correlation information between consecutive frames. However, such a pixel-level correlation is extracted at the cost of high computational complexity and large storage resource. In this paper, we propose a novel temporal feature extraction method, Attentive Correlated Temporal Feature (ACTF), by exploring inter-frame correlation within a certain region. The proposed ACTF exploits both bilinear and linear correlations between successive frames on the regional level. Our method has the advantage of achieving performance comparable to or better than optical flow-based methods while avoiding the introduction of optical flow. Experimental results demonstrate our proposed method achieves the competitive performances of 96.3% on UCF101 and 76.3% on HMDB51 benchmark datasets.