SYSTEMS AND METHODS FOR AUTO-SCALING A BIG DATA SYSTEM

Abstract Systems and methods for automatically scaling a big data system are disclosed. Methods may include: determining, at a first time, a first optimal number of nodes for a cluster to adequately process a request; assigning an amount of nodes equal to the first optimal number; determining a rate of progress of the request; determining, at a second time based on the rate of the progress a second optimal number of nodes; and modifying the number of nodes assigend to the cluster to equal the second optimal number. Systems may include: a cluster manager, to add and/or remove nodes; a big data system, to process requests that utilize the cluster and nodes, and an automatic scaling cluster manager, including: a big data interface, for comunnicating with the big data system; a cluster manager interface, for communicationg with a cluster manager instructions for adding and/or removing nodes from a cluster used to process a request; and a cluster state machine.