REAL TIME OPTIMIZATION OF COMPUTE INFRASTRUCTURE IN A VIRTUALIZED ENVIRONMENT

A system and method is provided to dynamically optimize the topography of a compute cluster in real time based on the runtime configuration, specified as metadata, of jobs in a queuing or scheduling environment. The system provisions or terminates multiple types of virtualized resources based on the profile of all applications of the jobs in a current queue based on their aggregate runtime configuration specified as requirements and rank expressions within associated metadata. The system will continually request and terminate compute resources as jobs enter and exit the queue, keeping the cluster as minimal as possible while still being optimized to complete all jobs in the queue, optimized for cost, runtime or other metric. End users can specify the runtime requirements of jobs, thereby preventing the user from having to know about the physical profile of the compute cluster to specifically architect their jobs.