METHOD AND SYSTEM FOR SEMANTIC APPEARANCE TRANSFER USING SPLICING ViT FEATURES

Using a pre-trained and fixed Vision Transformer (ViT) model as an external semantic prior, a generator is trained given only a single structure/appearance image pair as input. Given two input images, a source structure image and a target appearance image, a new image is generated by the generator in which the structure of the source image is preserved, while the visual appearance of the target image is transferred in a semantically aware manner, so that objects in the structure image are "painted" with the visual appearance of semantically related objects in the appearance image. A self-supervised, pre-trained ViT model, such as a DINO-VIT model, is leveraged as an external semantic prior, allowing for training of the generator only on a single input image pair, without any additional information (e.g., segmentation/correspondences), and without adversarial training. The method may generate high quality results in high resolution (e.g., HD).