Quantitative performance metrics for evaluation and comparison of middleware interoperability products

Interoperability in modeling and simulation is understood as the ability of simulations to exchange information and use the information exchanged. Several alternative models have been applied to support interoperability. The traditional approach has been to apply standards, such as the High-level Architecture, to create a federation and use a runtime infrastructure to physically connect simulations. Recently, there has been a move towards web-based standards to loosely couple simulations, and the future points to cloud-enabled interoperability services. Despite the existence of implementations of these models from industry, academia and government, very few performance metrics have been formulated to evaluate them. In this paper, we propose quantitative performance metrics that includes a set of dependent and independent variables for implementations of interoperability models. We apply the metrics to a web-based infrastructure that uses web standards and a traditional runtime infrastructure. We analyze the results and discuss the tradeoffs that federation developers have to consider when selecting an implementation of an interoperability model.