LOAD BALANCING USING PROGRESSIVE SAMPLING BASED ON LOAD BALANCING QUALITY TARGETS

A method, system, and computer program product for of load balancing in a parallel map/reduce paradigm. The method commences by sampling a first set of input records, and forming a prospective load balancing assignment by assigning the first set of input records to the plurality of worker tasks based on a workload estimate for each of the worker tasks. To measure the prospective load balancing assignment, the method compares the workload variance over the plurality of worker tasks to a workload variance target, and also calculates a confidence level based on the sampled first set of input records. If the measured quality of the prospective load balancing assignment is not yet achieved, then the method samples additional input records; for example when the calculated workload variance is greater than the maximum workload variance target or when the calculated confidence level is lower than a confidence level threshold.