Assessing and optimizing the performance impact of the just-in-time configuration parameters - a case study on PyPy

Many modern programming languages (e.g., Python, Java, and JavaScript) support just-in-time (JIT) compilation to speed up the execution of a software system. During runtime, the JIT compiler translates the frequently executed part of the system into efficient machine code, which can be executed much faster compared to the default interpreted mode. There are many JIT configuration parameters, which vary based on the programming languages and types of the jitting strategies (method vs. tracing-based). Although there are many existing works trying to improve various aspects of the jitting process, there are very few works which study the performance impact of the JIT configuration settings. In this paper, we performed an empirical study on the performance impact of the JIT configuration settings of PyPy. PyPy is a popular implementation of the Python programming language. Due to PyPy’s efficient JIT compiler, running Python programs under PyPy is usually much faster than other alternative implementations of Python (e.g., cPython, Jython, and IronPython). To motivate the need for tuning PyPy’s JIT configuration settings, we first performed an exploratory study on two microbenchmark suites. Our findings show that systems executed under PyPy’s default JIT configuration setting may not yield the best performance. Optimal JIT configuration settings vary from systems to systems. Larger portions of the code being jitted do not necessarily lead to better performance. To cope with these findings, we developed an automated approach, ESM-MOGA, to tuning the JIT configuration settings. ESM-MOGA, which stands for effect-size measure-based multi-objective genetic algorithm, automatically explores the PyPy’s JIT configuration settings for optimal solutions. Case studies on three open source systems show that systems running under the resulting configuration settings significantly out-perform (5% - 60% improvement in average peak performance) the default configuration settings.