Video-based face recognition and image synthesis from rotating head frames using nonlinear manifold learning by neural networks

This paper proposes a new video-based face recognition method that uses video frames of a subject rotating his/her head. In the experiment discussed here, the manifolds of video frames embedded in a high-dimensional video space were extracted using neural network (NN)-based models. This increased the recognition rate in comparison with a simple NN architecture (from 72.9 to 81.3 %). These models were inspired by manifold interpretations of the brain’s visual perception. Next, the pose and person manifolds were separated using the neurons trained in the hidden or bottleneck layer of the network. Finally, the separated manifolds were used to synthesize face images at different angles from a single frontal image. Using video frames to extract these manifolds produces higher-quality images than using manifolds extracted from single images. This improvement in image quality was verified using the structural similarity index measure.