Spatio-Temporal Deformable DETR for Weakly Supervised Defect Localization

Currently, the welding process is an essential part of various industrial fields, such as shipbuilding, automobiles, aerospace, etc. Since welding defects can lead to serious adverse consequences, they must be carefully monitored. With captured welding images, welding defect localization can be formulated as a prediction of the defective area bounding box in the image, which can be solved through deep learning techniques. However, annotating the bounding box is required to train a deep learning network, which is costly and time consuming. As such, we propose a spatio-temporal deformable detection Transformer (STD-DETR), which requires only frame-level labels in the learning phase and localizes the welding defects in the inference phase. Weakly supervised learning is feasible in our task because the attention map can be obtained using a self-attention mechanism without any pixel-level or bounding box labels. STD-DETR is developed to extract not only spatial patterns but also temporal patterns with spatio-temporal attention. In addition, STD-DETR utilizes a deformable mechanism to calculate the attention to reduce computational complexity. Our proposed network is trained and evaluated on our custom welding defect video dataset captured by our manual data acquisition equipment. We demonstrate that the proposed STD-DETR outperforms other weakly supervised object localization models in welding defect localization and binary classification.