Universal multi-Source domain adaptation for image classification

•The hypothesis of universal domain adaptation is violated in the multi-source setting.•Existing multi-source domain adaptation methods primarily focus on the closed set setting.•It is the target data that determines the common and private classes in the source domain.•Samples in the same class should share a common weight during class-wise alignment.•Model complexity should not increase with the change of domains. Unsupervised domain adaptation (DA) enables intelligent models to learn transferable knowledge from a labeled source domain and adapt to a similar but unlabeled target domain. Studies showed that knowledge could be transferred from one source domain to another unknown target domain, called Universal DA (UDA). However, there is often more than one source domain in the real-world application to be exploited for DA. In this paper, we formally propose a more general domain adaptation setting for image classification, universal multi-source DA (UMDA), where the label sets of multiple source domains can be different, and the label set of the target domain is completely unknown. The main challenge in UMDA is to identify the common label set among each source and target domain and keep the model scalable as the number of source domains increases. In the face of this challenge, we propose a universal multi-source adaptation network (UMAN) to solve the DA problem without increasing the complexity of the model in various UMDA settings. In UMAN, the reliability of each known class belonging to the common label set is estimated via a novel pseudo-margin vector and its weighted form, which helps adversarial training better align the distributions of multiple source domains and target domain. Moreover, the theoretical guarantee for UMAN is also provided. Massive experimental results show that existing UDA and multi-source DA (MDA) methods cannot be directly deployed to UMDA, and the proposed UMAN achieves the state-of-the-art performance in various UMDA settings.