Gesture sequence recognition with one shot learned CRF/HMM hybrid model

In this paper, we propose a novel markovian hybrid system CRF/HMM for gesture recognition, and a novel motion description method called gesture signature for gesture characterisation. The gesture signature is computed using the optical flows in order to describe the location, velocity and orientation of the gesture global motion. We elaborated the proposed hybrid CRF/HMM model by combining the modeling ability of Hidden Markov Models and the discriminative ability of Conditional Random Fields. In the context of one-shot-learning, this model is applied to the recognition of gestures in videos. In this extreme case, the proposed framework achieves very interesting performance and remains independent from the moving object type, which suggest possible application to other motion-based recognition tasks. •A hybrid CRF/HMM system for gesture recognition is proposed.•HMM and CRF advantages combination and disadvantages compensation.•Gesture Signature: an optical-flow-based gesture characterization model is proposed.•Evaluation on the Chalearn competition data set under a one-shot learning context.