RTM-UAVDet: A Real-Time Multimodal UAV Detector

As unmanned aerial vehicles (UAVs) become more prevalent, the need for accurate and reliable detection algorithms becomes increasingly important. UAV detection algorithms are essential for maintaining security, managing airspace, safeguarding privacy, responding to emergencies, protecting critical infrastructure, ensuring regulatory compliance, and enabling responsible UAV usage across various domains. The multi-scene images and videos based on visible light (RGB) and thermal infrared (TIR) remote sensing of UAVs are considered crucial data sources for public safety. However, as imaging techniques move from single to multimodality, using object detection algorithms to detect RGB and TIR modalities simultaneously in real-time is a significant challenge. This study proposes a multimodal UAV object detection framework for static images. Based on convolutional neural network (CNN) architecture, the multimodal dynamic convolution model extracts features from ground TIR images and videos of forward-looking infrared (FLIR) and visible light cameras. Our approach circumvents the constraints of conventional multimodal fusion methods by effectively detecting UAVs within single-frame, multimodal static images without requiring image alignment. The results show that the average precision 0.5:0.95 (AP_{0.5:0.95}) of the UAV instance in the validation task is 66.1%. Furthermore, RTM-UAVDet operates at a speed of 72.4 frames per second (FPS), fully satisfying the requirements for real-time processing and effectively recognizing UAVs larger than 48 (6×8) pixels.