Empirical evaluation of mutation‐based test case prioritization techniques

Summary In this paper, we propose a new test case prioritization technique that combines both mutation‐based and diversity‐aware approaches. The diversity‐aware mutation‐based technique relies on the notion of mutant distinguishment, which aims to distinguish one mutant's behaviour from another, rather than from the original program. The relative cost and effectiveness of the mutation‐based prioritization techniques (i.e., using both the traditional mutant kill and the proposed mutant distinguishment) are empirically investigated with 352 real faults and 553,477 developer‐written test cases. The empirical evaluation considers both the traditional and the diversity‐aware mutation criteria in various settings: single‐objective greedy, hybrid, and multi‐objective optimization. The results show that there is no single dominant technique across all the studied faults. To this end, the reason why each one of the mutation‐based prioritization criteria performs poorly is discussed, using a graphical model called Mutant Distinguishment Graph that demonstrates the distribution of the fault‐detecting test cases with respect to mutant kills and distinguishment. © 2018 John Wiley & Sons, Ltd. The relative cost and effectiveness of mutation‐based test case prioritization techniques are empirically investigated with 352 real faults and 553,477 developer‐written test cases. The results show that there is no single dominant technique across all the studied faults. To this end, the reason why each one of the mutation‐based prioritization criteria performs poorly is discussed, using a graphical model called Mutant Distinguishment Graph that demonstrates the distribution of the fault‐detecting test cases with respect to mutant kills and distinguishment.