Cause reduction: delta debugging, even without bugs

Summary What is a test case for? Sometimes, to expose a fault. Tests can also exercise code, use memory or time, or produce desired output. Given a desired effect, a test case can be seen as a cause, and its components divided into essential (required for effect) and accidental. Delta debugging is used for removing accidents from failing test cases, producing smaller test cases that are easier to understand. This paper extends delta debugging by simplifying test cases with respect to arbitrary effects, a generalization called cause reduction. Suites produced by cause reduction provide effective quick tests for real‐world programs. For Mozilla's JavaScript engine, the reduced suite is possibly more effective for finding faults. The effectiveness of reduction‐based suites persists through changes to the software, improving coverage by over 500 branches for versions up to 4 months later. Cause reduction has other applications, including improving seeded symbolic execution, where using reduced tests can often double the number of additional branches explored. Copyright © 2015 John Wiley & Sons, Ltd. This paper presents a generalization of delta debugging (cause reduction) in which test cases are reduced based on an arbitrary property, rather than only with respect to ability to trigger a fault. The graph shows that basing test suites on tests cases that have been reduced with respect to statement coverage can result in much better code coverage for very small regression tests than using unreduced test cases. Cause reduction can also benefit symbolic execution and stress testing.