Robust one-stage object detection with location-aware classifiers

•We analyze the limitation of the classification head in one-stage detectors, which fills the gap in the literature.•We explain the classifier's limitation by visualizing its representations and analyzing its robustness to the scene context.•The findings give insights to design location-aware multi-dilation module (LAMD) in the classifiers for robust detection.•Experiments on MS COCO across various detectors with different backbones show that our method can achieve higher performance. Recent progress on one-stage detectors focuses on improving the quality of bounding boxes, while they pay less attention to the classification head. In this work, we focus on investigating the influence of the classification head. To understand the behavior of the classifier in one-stage detectors, we resort to the methods of the Explainable deep learning area. We visualize its learned representations via activation maps and analyze its robustness to image scene context. Based on the analysis, we observe that the classifier limits the performance of the detector due to its limited receptive field and the lack of object locations. Then, we design a simple but efficient location-aware multi-dilation module (LAMD) to enhance the weak classifier. We conduct extensive experiments on the COCO benchmark to validate the effectiveness of LAMD. The results suggest that our LAMD can achieve consistent improvements and leads to robust detection across various one-stage detectors with different backbones.