Three-dimensional Entity Resolution with JedAI

Entity Resolution (ER) is the task of detecting different entity profiles that describe the same real-world objects. To facilitate its execution, we have developed JedAI, an open-source system that puts together a series of state-of-the-art ER techniques that have been proposed and examined independently, targeting parts of the ER end-to-end pipeline. This is a unique approach, as no other ER tool brings together so many established techniques. Instead, most ER tools merely convey a few techniques, those primarily developed by their creators. In addition to democratizing ER techniques, JedAI goes beyond the other ER tools by offering a series of unique characteristics: (i) It allows for building and benchmarking millions of ER pipelines. (ii) It is the only ER system that applies seamlessly to any combination of structured and/or semi-structured data. (iii) It constitutes the only ER system that runs seamlessly both on stand-alone computers and clusters of computers — through the parallel implementation of all algorithms in Apache Spark. (iv) It supports two different end-to-end workflows for carrying out batch ER (i.e., budget-agnostic), a schema-agnostic one based on blocks, and a schema-based one relying on similarity joins. (v) It adapts both end-to-end workflows to budget-aware (i.e., progressive) ER. We present in detail all features of JedAI, stressing the core characteristics that enhance its usability, and boost its versatility and effectiveness. We also compare it to the state-of-the-art in the field, qualitatively and quantitatively, demonstrating its state-of-the-art performance over a variety of large-scale datasets from different domains. The central repository of the JedAI’s code base is here: https://github.com/scify/JedAIToolkit . A video demonstrating the JedAI’s Web application is available here: https://www.youtube.com/watch?v=OJY1DUrUAe8. •JedAI is an open-source system that allows for composing the state-of-the-art individual methods into millions of end-to-end workflows for Entity Resolution.•JedAI supports both batch and progressive Entity Resolution.•JedAI supports both blocking-based and join-based Entity Resolution.•JedAI supports both serialized and massively parallel execution (on top of Apache Spark).•JedAI achieves comparable effectiveness to the state-of-the-art (supervised) ER tools at a significantly lower running time.