A comprehensive survey on automatic facial action unit analysis

Facial Action Coding System is the most influential sign judgment method for facial behavior, and it is a comprehensive and anatomical system which could encode various facial movements by the combination of basic AUs (Action Units). AUs define certain facial configurations caused by contraction of one or more facial muscles, and they are independent of interpretation of emotions. However, automatic facial action unit recognition remains challenging due to several open questions such as rigid and non-rigid facial motions, multiple AUs detection, intensity estimation and naturalistic context application. This paper introduces recent advances in automatic facial action unit recognition, focusing on the fundamental components of face detection and registration, facial action representation, feature selection and classification. The comprehensive analysis of facial representations is presented according to the facial data properties (image and video, 2D and 3D) and characteristics of facial features (predesign and learning, appearance and geometry, hybrid and fusion). Facial action unit recognition involves AUs occurrence detection, AUs temporal segment detection and AUs intensity estimation. We discussed the role of each component, main techniques with their characteristics, challenges and potential directions of facial action unit analysis.