Testing of Heterogeneous Systems

Context: A system of systems often exhibits heterogeneity, for instance in implementation, hardware, process and verification. We define a heterogeneous system, as a system comprised of multiple systems (system of systems) where at least one subsystem exhibits heterogeneity with respect to the other systems. The system of systems approach taken in development of heterogeneous systems give rise to various challenges due to continuous change in configurations and multiple interactions between the functionally independent subsystems. The challenges posed to testing of heterogeneous systems are mainly related to interoperability, conformance and large regression test suites. Furthermore, the inherent complexities of heterogeneous systems also pose challenge to the specification, selection and execution of tests. Objective: The main objective of this licentiate thesis is to provide an insight on the state of the art in testing heterogeneous systems. Moreover, we also aimed to investigate different test techniques used to test heterogeneous systems in industrial settings and their usefulness as well as to identify and prioritize different information sources that can help practitioners to define a generic search space for test case selection process. Method: The findings presented in this thesis are obtained through a controlled experiment, a systematic literature review (SLR), a case study and an exploratory survey. The purpose of systematic literature review was to investigate the existing state of art in testing heterogeneous systems and identification of research gaps. The results from the SLR further laid down the foundation of action research conducted through an exploratory survey to compare different test techniques. We also conducted an industrial case study to investigate the relevant data sources for search space initiation to prioritize and specify test cases in context of heterogeneous systems. Results: Based on our literature review, we found that testing of heterogeneous systems is considered a problem of integration and system testing. It has been observed that multiple interactions between the system and subsystems results into a testing challenge, especially when the configurations change continuously. It is also observed that current literature targets the problem of testing heterogeneous systems with multiple test objectives resulting in employing different test methods to reach a domain specific testing challenge. Using the exploratory surve