An interpretive evaluation of entity summarization system

The task of entity summarization (ES) is to select an optimum subset from a large set of triples describing an entity in a knowledge graph.ES systems often integrate many and various ES features in a complex way.While state-of-the-art ES systems have been evaluated and compared by recent benchmarking efforts, it was unclear whether and how much each constituent ES feature had contributed to the performance of an ES system.An interpretive evaluation of ES systems was proposed.Two novel evaluation metrics were proposed, feature effectiveness ratio and feature significance ratio, to characterize how much ground-truth summaries and machine-generated summaries exhibit each ES feature.Their comparison would help to interpret the performance of an ES system.Based on three benchmarks, metrics with six popular ES features were implemented, and an interpretive evaluation of nine unsupervised ES systems and two supervised ES systems were presented.The code and data are open source.