Processing high data rate streams in System S

High-performance stream processing is critical in many sense-and-respond application domains—from environmental monitoring to algorithmic trading. In this paper, we focus on language and runtime support for improving the performance of sense-and-respond applications in processing data from high-rate live streams. The central tenets of this work are the programming model, the workload splitting mechanisms, the code generation framework, and the underlying System S middleware and Spade programming model. We demonstrate considerable scalability behavior coupled with low processing latency in a real-world financial trading application. ► The split/aggregate/join architectural pattern is identified as a common template for implementing stream processing applications in different domains. ► Per-group processing is proven as an effective means of optimizing the implementation of the split/aggregate/join pattern. ► System S and SPADE are employed to provide a comprehensive performance evaluation of per-group split/aggregate/join processing on a cluster of machines, using a financial engineering application as a case study.