A novel facial emotion recognition method for stress inference of facial nerve paralysis patients

•We built an emotional face video dataset from facial nerve paralysis patients.•The emotions of facial nerve paralysis patients are recognizable via face images.•Transfer learning helps conquer the problem of limited data size in the clinic.•It is feasible to infer emotional stress state of a facial nerve paralysis patient.•Emotion recognition and stress inference may help improve emotional well-being. Facial nerve paralysis results in muscle weakness or complete paralysis on one side of the face. Patients suffer from difficulties in speech, mastication and emotional expression, impacting their quality of life by causing anxiety and depression. The emotional well-being of a facial nerve paralysis patient is usually followed up during and after treatment as part of quality-of-life measures through questionnaires. The commonly used questionnaire may help recognize whether a patient has been through a depressive state but is unable to understand their basic emotions dynamically. Automatic emotion recognition from facial expression images could be a solution to help understand facial nerve paralysis patients, recognize their stress in advance, and assist their treatment. However, their facial expressions are different from healthy people due to facial muscle inability, which makes existing emotion recognition data and models from healthy people invalid. Recent studies on facial images mainly focus on the automatic diagnosis of facial nerve paralysis level and thus lack full basic emotions. Different nerve paralysis levels also increase inconsistency in expressing the same emotion among patients. To enable emotion recognition and stress inference from facial images for facial nerve paralysis patients, we established an emotional facial expressions dataset from 45 patients with six basic emotions. The problem of limited data size in building a deep learning model VGGNet was solved by leveraging facial images from healthy people in transfer learning. Our proposed model reached an accuracy of 66.58% recognizing basic emotions from patients, which was 19.63% higher than the model trained only from the facial nerve paralysis data and was 42.69% higher than testing directly on the model trained from healthy data. Logically, the results show that patients with less severe facial nerve paralysis reached a higher emotion recognition accuracy. Additionally, although disgust, anger, and fear were especially challenging to specify from each other, the accuracy was 85.97% r