Dragon Proxy Runtimes and Multi-system Workflows

We present a novel method for obtaining proxy access to remote instances of the Dragon distributed runtime. Dragon is a composable distributed runtime for managing dynamic processes, high-performance communication objects, memory and data at scale that is based on an abstraction of a distributed system. Proxy access to a remote instance of the Dragon runtime allows the client Dragon runtime to run any command that could be run directly by the remote Dragon runtime, but executes the command on the remote runtime. Commands to be run on a remote Dragon runtime are mediated by a Python object that acts as a proxy for the remote runtime, which we call a \textit{proxy runtime}. These proxy runtimes, combined with the ability to start and tear down remote Dragon runtimes both programmatically and via the command line interface, make a number of challenging workflows simple to program. Such workflows include edge-to-cloud scientific workflows, batch services and scientific applications based on Python multiprocessing. The ability to program complex workflows on systems that span clusters, scientific instruments and cloud resources is critical to the development of post-exascale applications, infrastuctures and frameworks.