Ultra-lightweight face activation for dynamic vision sensor with convolutional filter-level fusion using facial landmarks

[Display omitted] •Design hand-craft filters for facial feature extraction from landmark data.•Develop an ultra-lightweight face activation network using landmark filters.•Achieve feasible performance and computational cost in a mobile environment.•Construct dataset for face activation obtained from a dynamic vision sensor. As dynamic vision sensors can operate at low power while having a fast response, they can mitigate the disadvantages of gyro sensors when used for turning on mobile devices. Therefore, we propose an ultra-lightweight face activation neural network that combines handcrafted convolutional landmark filters extracted from facial features with randomly initialized trainable convolutional filters. Face activation is the task of identifying the presence or absence of a face intended to activate the mobile device. Our proposed model, F-LandmarkNet, has four steps. First, we construct customized landmark filters that can effectively identify numerous facial features. Second, F-LandmarkNet is constructed by using a convolutional layer that fuses handcrafted landmark filters and trainable convolution filters. Third, a compact version is constructed by selecting only the four most influential face filters according to their importance. Finally, performance is improved through knowledge distillation. The fusion of handcrafted landmark filters and trainable convolutional filters is quite effective in extremely lightweight models. It is observed that the classification accuracy of our proposed model is similar to that of existing lightweight convolutional neural network models, while the number of floating-point operations and parameters are markedly lower. Our model also runs faster under a central processing unit environment than comparison models. Thus, the proposed model shows high potential for use in actual mobile systems.