Optical flow and scene flow estimation: A survey

•To the best of our knowledge, this is the first survey paper to cover both optical flow and scene flow estimation comprehensively.•In contrast to existing surveys, this survey includes knowledge-driven, data-driven and hybrid-driven methods.•Comprehensive comparisons of existing methods on several datasets are provided with insightful observations and sufficient analyses.•Some of the open issues and potential research directions are discussed. Motion analysis is one of the most fundamental and challenging problems in the field of computer vision, which can be widely applied in many areas, such as autonomous driving, action recognition, scene understanding, and robotics. In general, the displacement field between subsequent frames can be divided into two types: optical flow and scene flow. The optical flow represents the pixel motion of adjacent frames. In contrast, the scene flow is a 3D motion field of the dynamic scene between two frames. Traditional approaches for the estimation of optical flow and scene flow usually leverage the variational technique, which can be solved as an energy minimization process. In recent years, deep learning has emerged as a powerful technique for learning feature representations directly from data. It has led to remarkable progress in the field of optical flow and scene flow estimation. In this paper, we provide a comprehensive survey of optical flow and scene flow estimation. First, we briefly review the pioneering approaches that use variational technique and then we delve in detail into the deep learning-based approaches. Furthermore, we present insightful observations on evaluation issues, specifically benchmark datasets, evaluation metrics, and state-of-the-art performance. Finally, we give the promising directions for future research. To the best of our knowledge, we are the first to review both optical flow and scene flow estimation, and the first to cover both traditional and deep learning-based approaches.