Multimodal Unsupervised Domain Adaptation for Predicting Speaker Characteristics from Video

Persuasion and expertise are two central skills of a speaker. In this work, we propose a multimodal unsupervised domain adaptation method to predict the persuasiveness and expertise of the speaker in a video. The proposed approach uses MAG-BERT for modeling the multimodal feature space and the adversarial discriminative domain adaptation approach, which is a generalized approach for unsupervised domain adaptation. To reduce the domain shift, knowledge distillation is added to the adversarial adaptation. We explore two methods in an adversarial framework; the standard generative adversarial networks (GAN) loss and the Wasserstein Generative Adversarial Network (WGAN) loss, and we also explore two methods for incorporating more domain knowledge: knowledge distillation and probabilistic knowledge transfer. The experiments demonstrate that the proposed approach is promising for predicting the persuasiveness and expertise of speakers in videos using multimodal data. The best set of results for persuasiveness prediction using multimodal data is with the MAG-BERT WGAN model with an accuracy of 68% and an f1-score of 0.81 and with the BERT WGAN model with an accuracy of 75% and an f1-score of 0.81. In the case of expertise prediction, the MAG-BERT WGAN model resulted in an accuracy of 57% and an f1-score of 0.63.