Towards the adversarial robustness of facial expression recognition: Facial attention-aware adversarial training

Beyond the in-the-lab environment, deep-learning-based facial expression recognition (FER) models that provide reliable performance on wild datasets are gradually becoming applied to the real world. However, the fact that neural networks are inherently vulnerable to digital attacks (e.g., adversarial examples) and their performance is not exposed to external threats reduces the applicability of FER technology. So, we design a so-called test-time attack scenario in which FER models are deceived by superimposing imperceptible perturbation(s) on test images. This scenario, which targets the testing phase in which model weakness is revealed, clearly shows how vulnerable FER models are to external attacks. As a remedy against this attack, we propose a novel method called FAAT, which adversarially trains the model by paying attention to core region(s) of face. FAAT aims to improve model robustness so that the model can be generalized to unseen perturbation(s) while focusing on facial expression-related areas. For example, FAAT’s robustness against PGD attack with a performance improvement of up to 18% is encouraging. Also, various benchmarking results based on our attack scenario analyze the fidelity of prior arts and will promote the development direction of future models. •Imperceptible perturbations that can occur at test time in applications using facial expression recognition (FER).•Create a landmark-based mask specialized for face-related tasks and use it for adversarial training.•Gives the model a purification effect through regularization using frequency-based loss.•Superior defense performance not only in emotion classification FER but also in VA regression FER.