A simulation-based approach to determine the evaluation adequacy of system-of-systems operational test configurations

A system-of-systems is defined as “a set or arrangement of systems that results from independent systems integrated into a larger system that delivers unique capabilities.” Given practical resource constraints, it is rare that the full-field configuration of the system-of-systems can be exercised during an operational reliability demonstration test. However, as we consider various potential operational test configurations for a given system-of-systems during the reliability test program planning process, it is critical to understand how testing a configuration that is smaller than the full-field configuration decreases the adequacy of the test by reducing the accuracy of the system-of-systems’ reliability estimate that is based on the test results. Thus, it is useful to assess the adequacy of potential system-of-systems’ operational test configurations before adopting one. We present a novel simulation-based method that can be employed to assess the adequacy of a given test configuration for any type of system-of-systems. To illustrate how this simulation-based method can be used to aid in the identification of the best alternative from among a group of potential operational test configuration alternatives, we include an example application using a notional air defense system-of-systems. Trade-offs with respect to cost, schedule, and accuracy are addressed within the context of this application.