Modeling swarm mission with COTS characterization: a series of return on experience

System design in defense systems is a competitive field, in which economical viability relies on a sequence of architectural decisions, aiming at quality, resource and time (Q,R,T) compromises. We observe that low‐cost unmanned ground vehicles (UGV) and drones appear as new threats on current battlefield. To face these new threats, Direction Générale de l'Armement (DGA) have organized challenges around robotization of battlefield, to design future employment doctrines and help technologies to reach maturity in a reasonable time. This article exposes a NATO Architecture Framework (NAF) 3.1‐based workflow that includes return of experience form the field over yearly iterations of such challenges. The capabilities depicted are requirements to match, constituent systems are based on Components‐Off‐The‐Shelf (COTS) answering to both edition of the challenge. This article details how manually re‐injecting feedback from field back to the system model failed to ensure the next iterations of the challenge. Our works propose conclusions on formulation of the “engineering leakage problem” and how resolution of this problem is NP‐Hard and should be addressed using optimization.