3-D Facial Priors Guided Local-Global Motion Collaboration Transforms for One-Shot Talking-Head Video Synthesis

This paper presents a method that takes a sequence of audio, pose source video, and one reference image as input to generate natural head spatial movement talking face video. Based on the powerful 3D Morphable Model(3DMM), we propose a novel framework comprising two modules: Audio Map Expression Net and Local-Global Motion Collaboration Transforms Render Net. To make the audio map expression network predict more precise 3D facial expression parameters, we propose to use the shape constraint loss and the lip-sync constraint loss as additional constraints. The proposed Local-Global Motion Collaboration Transforms Render Net aims to improve generalization capability in the presence of apparent head motion by utilizing cross-modal images (3D reconstruction face and realistic face) as input to predict the residual motion of pose-aware dynamic content. Our work demonstrates the ability of Convolutional Neural Networks to utilize cross-modal images as input for simultaneous estimation of pose-aware dynamic content and local residual motion through self-supervised learning. Finally, we propose a multi-scale feature adaptive denormalization (FADE) U-net network architecture to exploit coarse-to-fine feature representations to improve the generated image quality. Experimental results demonstrate the effectiveness compared to other state-of-the-art methods.