System and method for creating on-demand virtual filesystem having virtual burst buffers created on the fly

A method for creating a virtual intermediary filesystem on the fly and in real-time on a compute node cluster having a plurality of compute nodes that are running or scheduled to run an application job is provided The method includes initializing a data structure, allocating a plurality of compute nodes, reserving the compute node cluster, distributing the application job, determining memory utilization, determining potential free memory space, marking the potential free memory space, and mapping the marked potential free memory space to create the virtual intermediary filesystem on-demand and in real-time, wherein the virtual intermediary filesystem is arranged to store data in parallel and concurrently from a plurality of computing devices on the compute node cluster and output the stored data to a storage system, wherein the computing devices output the data at a rate that is greater than a maximum write or read data rate of the storage system.