Fast and accurate image retrieval using knowledge distillation from multiple deep pre-trained networks

The content retrieval systems aim to retrieve images similar to a query image from a large data set. A feature extractor and similarity measure play a key role in these systems. Hand-crafted feature descriptors like SURF, SIFT, and GIST find a suitable pattern for measuring the similarity between images. Recently deep learning in this field has been given much attention, which performs feature extraction and similarity learning simultaneously. Various research shows that the feature vector extracted from pre-trained networks contains richer information than class labels in classifying or retrieving information. This paper presents an effective method, Deep Muti-teacher Transfer Hash (DMTH), which uses knowledge from several complex models to teach a simple one. Due to the variety of available pre-trained models and the diversity among their extracted features, we utilize an attention mechanism to obtain richer features from them to teach a simple model via an appropriate knowledge distillation loss. We test our method on widely used datasets Cifar10 & Cifar100 and compare our method with other state-of-the-art methods. The experimental results show that DMTH can improve the image retrieval performance by learning better features obtained through an attention mechanism from multiple teachers without increasing evaluation time. Specifically, the proposed multi-teacher model surpasses the best individual teacher by 2% in terms of accuracy on Cifar10. Meanwhile, it boosts the performance of the student model by more than 4% using our knowledge transfer mechanism.