SOFA: An extensible logical optimizer for UDF-heavy data flows

Recent years have seen an increased interest in large-scale analytical data flows on non-relational data. These data flows are compiled into execution graphs scheduled on large compute clusters. In many novel application areas the predominant building blocks of such data flows are user-defined predicates or functions (Udfs). However, the heavy use of Udfs is not well taken into account for data flow optimization in current systems. Sofa is a novel and extensible optimizer for Udf-heavy data flows. It builds on a concise set of properties for describing the semantics of Map/Reduce-style Udfs and a small set of rewrite rules, which use these properties to find a much larger number of semantically equivalent plan rewrites than possible with traditional techniques. A salient feature of our approach is extensibility: we arrange user-defined operators and their properties into a subsumption hierarchy, which considerably eases integration and optimization of new operators. We evaluate Sofa on a selection of Udf-heavy data flows from different domains and compare its performance to three other algorithms for data flow optimization. Our experiments reveal that Sofa finds efficient plans, outperforming the best plans found by its competitors by a factor of up to six. •We identify a small set of UDF properties crucial for data flow optimization.•SOFA employs taxonomy and reasoning to enhance extensibility of data flow languages.•SOFA is capable of rewriting DAG-shaped data flows given proper operator annotations.•We evaluate our approach using a diverse set of data flows across different domains.•SOFA finds more efficient plans compared to existing data flow optimizers.•Optimization with SOFA is even more beneficial when working on very large data sets.