Revisiting the Generalization Ability of Deep Local Descriptors

Recently, learned local descriptors based on convolutional neural networks have achieved promising performance on standard benchmarks, yet still trailed the generalization ability of hand-crafted descriptors. In this paper, we analyze the reasons and mitigate the problem by promoting the learning of cross-domain discriminative features. Specifically, it is discovered that a vast number of easy positives are fed to the triplet margin loss during training, which can overwhelm the learning of informative hard negatives. To deal with it, we propose the Adaptive Positive Weighting Loss (APWL), which reshapes the triplet margin loss with a hardness-aware scaling factor to down-weight the contribution of easy positives and focus the model on hard negatives for learning discriminative features. Moreover, the domain discrepancy frequently encountered in the real world is observed to collapse the feature space of the target domain and impair the discriminative ability of learned descriptors. Therefore, a Cross-domain Distribution Regulation (CDR) is introduced to unfold the collapsed feature space so that discriminative features can be retained across domains to further boost the generalization ability of learned descriptors. Extensive experimental results demonstrate that when trained with the proposed loss and regulation term, the learned descriptors can be empowered with stronger generalization ability and outperform state-of-the-art methods on standard benchmarks.