Design and development of a DDDAMS-based border surveillance system via UVs and hybrid simulations

•An effective and efficient control framework for UAV/UGV coordination is proposed.•The detection and classification algorithms are developed to process real-time data.•Physics-based simulation is utilized to provide uncertainties for robust planning and control of UVs.•Target's proactive behaviors are represented based on utility-based decision-making model.•The framework is demonstrated for a hardware-in-the-loop simulation environment. Despite the increasing use of sensor technologies in border surveillance applications, there is a lack of systematic methodology and its implementation with the effective control system. The challenge arises due to information heterogeneity and uncertainty caused by the usage of different sensors. This paper extends the authors’ previous dynamic-data-driven adaptive multi-level simulation (DDDAMS) framework to overcome this challenge using unmanned vehicles (UVs), sensors, and multi-level simulation. Specifically, the detection and classification algorithms are employed to process real-time data generated by fixed (e.g. geophone) and mobile (e.g. UV camera) sensors for the adequate monitoring. Also, physics-based simulation (PBS) is utilized to provide uncertainties for robust planning and control of UVs. The environmental effects as well as target's proactive behavior against the surveillance system are incorporated into the framework using utility-based decision-making model. Finally, we provide a detailed description regarding field demonstration, where autonomous control of UVs and real-time communications among all system components are attained through PBS.