Facial expression recognition in facial occlusion scenarios: A path selection multi-network

The occlusion scenarios including wearing mask, wearing sunglasses, wearing hat, etc. Thus, parts of face are occluded, e.g. nose and mouth are obscured if a mask is worn. Facial expression recognition (FER) tasks have been widely researched. However, less attention has been paid to FER in occlusion scenarios, which are not uncommon in the real world. In this paper, we propose a method that structures a path selection multi-network model to achieve the FER in the above three types of facial occlusion scenarios. The method contains two parts. For the multi-network, we segment the labels in one database which results in three new sub-databases to train three Subnets, respectively. For the path selection, which is an integration method of multi-network, we merge groups of labels in one database to train an initial network called BeginNet. The prediction of BeginNet selects one of the Subnets to make the final prediction. We concatenate four popular databases Fer2013, JAFFE, KDEF and Raf-DB databases into one larger database, and use the combined database to verify our method effectiveness. The experimental results show that our method has better results in coping with the expression recognition task in multiple types of facial occlusion scenarios. •Recognize facial expression in facial occlusion scenarios.•Each sub-network in the proposed multi-network only is used to classify a part of facial expressions.•A new integration method is proposed for the multi-network structures.•The proposed method of facial expression recognition can deal with multiple types of facial occlusion scenarios.