Scale-balanced loss for object detection

•A matching imbalance in current object detection pipelines is pointed out. It can lead to poor performance of detecting objects with different scales.•An innovative loss function called scale-balanced loss is proposed to alleviate the matching imbalance.•Experiments demonstrate the effectiveness of the scale-balanced loss, especially the performance of detecting small objects is improved significantly. Object detection is an important field in computer vision. Nevertheless, a research area that has so far not received much attention is the study into the effectiveness of anchor matching strategy and imbalance in anchor-based object detection, in particular small object detection. It is clear that the objects with larger sizes tend to match more anchors than smaller ones. This matching imbalance may result in poor performance in detecting small objects. It can be alleviated by paying more attention to the objects that match with fewer anchors. We propose an innovative flexible loss function for object detection, which is compatible with popular anchor-based detection methods. The proposed method, called the scale-balanced loss, does not add any extra computational cost to the original pipelines. By re-weighting strategy, the proposed method significantly improves the accuracy of multi-scale object detection, especially for small objects. Comprehensive experiments indicate that the scale-balanced loss achieved excellent generalization performance when separately applied to some popular detection methods. The scale-balanced loss attained up to 15% improvements on recall rates of small and medium objects in both the PASCAL VOC and MS COCO dataset. It is also beneficial to the AP result on MS COCO with an improvement of more than 1.5%.