Challenges and Techniques for Reproducible Simulations

Too often, reproducibility is unnecessarily sacrificed in new simulation codes. We explore some ways in which this happens and provide recommendations for reclaiming it. Experience shows that robust bitwise reproducibility on a fixed runtime platform is a desirable and achievable target. The variety of threats considered suggests that maintaining a reproducible simulator to this degree requires vigilance, but, in addition to the usual benefits, the increased effort is rewarded on the software engineering front by enabling low-overhead techniques to detect bugs sooner and diagnose them faster.