Recognition of 3D emotional facial expression based on handcrafted and deep feature combination

•Deep and handcrafted features are extracted from 2D depth images and 3D scans respectively.•The extracted features are embedded into a covariance pooling for the dimensionality reduction.•Deep covariance learning is carried out using two additional layers to enhance their discrimination power.•The Gaussian kernel is used to generalize the radial basis function to the manifold space for SVM classification.•High classification performance is achieved on two challenging datasets. Facial emotion recognition (FER) methods have been proposed mainly using 2D images. These methods suffer from many problems caused by the difficult conditions of unconstrained environments such as light conditions and view variations. In this paper, we aim to recognize the emotional facial expressions independently of their identity using the 3D data and 2D depth images. Since the 3D FER is a very fine-grained recognition task, mapping the 3D images into 2D depth images may lack some geometric characteristics of the expressive face and decay the FER performance. Convolutional Neural Networks (CNN), however, have been successfully applied to the 2D depth images and improved handcrafted-based methods in computer vision and pattern recognition applications. For this reason, we combine in this paper two types of features; handcrafted and deep learning features and prove their complementarity for 3D FER. Favorably, covariance descriptors have proven a very good ability to combine features from different types into a compact representation. Therefore, we propose to use the covariance matrices of features (handcrafted and deep ones), instead of the features independently. Since covariance matrices belong to one of the manifold space types, formed by SPD (Symmetric Positive Definite) matrices, we mainly focus on the generalization of the RBF kernel to the manifold space for 3D FER using a supervised SVM classification. The achieved performance of the proposed method on the Bosphorus and BU-3DFE datasets outperforms similar state-of-the-arts.