Reflections on the creation of a real-time parallel benchmark suite

Standard benchmark suites are a popular way to measure and compare computers performance. The Honeywell Technology Center has developed two benchmarking suites for parallel computers, the C3I Parallel Benchmark Suite (C3IPBS) and more recently, the Real Time Parallel Benchmark Suite (RTPBS). The C3IPB suite was similar to other benchmarks but focused on a new domain. The development of the real-time benchmark suite presented a number of new challenges. In this paper we share some of the lessons we have learned about the art of making parallel benchmarks.