Applying generic timing tests for distributed multimedia software systems

With recent advances in network technologies and computing power, multimedia systems have become a popular means for information delivery. However, testing of these systems is difficult. Due to incomplete control of their runtime and communication environment, precise temporal properties of multimedia systems are nonreproducible. Traditional software testing, which mainly deals with functional correctness, cannot be directly applied to testing temporal properties. Furthermore, time points are hard to be measured exactly, and in this sense are nondeterministic and nonreproducible. To address this problem, we propose a framework for testing the generic temporal properties of media objects in distributed multimedia software systems (DMSS). The timing properties are based on Allen's basic binary temporal relations between two objects, which can be extended to cover multiple objects. We have developed techniques for test case generation, and test result analysis based on a distributed tester architecture. Test templates are used in test case generation to reduce the possibility of human error, and the entire testing procedure can be automated. A prototype system has been built to test a DEC HPAS multimedia presentation system, which is a multimedia system supporting W3C's SMIL standard. Detailed discussions on practical issues illustrated with a number of actual tests are given. Experimental results have shown that our framework is effective in detecting errors in temporal properties. Furthermore, ways to reduce the test effort have been discussed, and guidelines for coming up with criteria for verdict computation based on the real-time requirements of the applications have been suggested.