Dynamic Facial Emotion Recognition Oriented to HCI Applications

As part of a multimodal animated avatar previously presented in Marcos-Pablos et al. ((2010) A realistic, virtual head for human-computer interaction. Interact. Comput., 22, 176–192, ISSN 0953-5438), in this paper we describe a method for dynamic recognition of displayed facial emotions on low-resolution streaming images. First, we address the detection of action units (AUs) of the facial action coding system using active shape models and Gabor filters. Normalized outputs of the AU recognition step are then used as inputs for a neural network that consists of an habituation network plus a competitive network. Both the competitive and the habituation layer use differential equations, thus taking into account the dynamic information of facial expressions through time. Experimental results carried out on live video sequences and on the Cohn-Kanade face database show that the proposed method provides high recognition hit rates. To assess the suitability of the developed emotional recognition system for human–computer interaction applications, it has been successfully integrated in the architecture of an avatar and we have conducted a preliminary experiment on empathy. The experiment showed promising results, as the avatar that made use of the emotional recognition system obtained a clear increase in the positivity of the rating when compared with the same avatar with no emotional response.