An exploratory empirical eye-tracker study of visualization techniques for coverage of combinatorial interaction testing in software product lines

Software Product Lines (SPLs) typically provide a large number of configurations to cater to a set of diverse requirements of specific markets. This large number of configurations renders unfeasible to test them all individually. Instead, Combinatorial Interaction Testing (CIT) computes a representative sample according to criteria of the interactions of features in the configurations. We performed an empirical study using eye-tracker technologies to analyze the effectiveness of two basic visualization techniques at conveying test coverage information of ten case studies of varying complexity. Our evaluation considered response accuracy, time-on-task, metacognitive monitoring, and visual attention. The study revealed clear advantages of a visualization technique over the other in three evaluation aspects, with a reverse effect depending on the strength of the coverage and distinct areas of visual attention. •Study of visualization techniques for interaction testing of Software Product Lines.•Analysis of visual attention using eye-trackers for coverage testing tasks.•Scatter plots and Parallel dimensions plots offer different performance trade-offs.