Mitigate the classification ambiguity via localization-classification sequence in object detection

•It is found that the classification ambiguity exists in both the inference and training processes to different degrees due to the non-correlation between classification and localization.•The proposed localization-classification sequence detector adopts the refined anchors as the intermediates to bridge the gap between the classification and localization, thereby mitigating the classification ambiguity.•The refinement-aware classification branch and assignment are proposed to perform refinement-aware classification and overcome corresponding obstacles.•The proposed method can mitigate the classification ambiguity to a large extent and achieve stable improvement across different baselines. In anchor-based detectors, the confidence scores and label-assignment results for the classification task are determined by the unrefined anchors rather than the final-refined boxes, which causes classification ambiguity due to the lack of correlation between the classification and localization tasks. In this paper, we investigate the classification ambiguity thoroughly via extensive experiments, and present the localization-classification sequence detector (LCSDet) that performs localization and classification in order, bridging the gap between them. To achieve this, the refinement-aware (RA) classification branch and RA assignment are proposed in LCSDet. In inference, the RA classification branch rectifies the feature misalignment and directly classifies the refined anchors. During training, the RA assignment tackles the training instability, narrows the location-quality gap and assigns the refined anchors to ground-truth objects. Comprehensive experiments indicate that the LCSDet can effectively mitigate the classification ambiguity and achieve stable improvement across different baselines.