Research on Hadoop Greedy Scheduler Based on the Fair

Job scheduling technology is one of the Hadoop platform’s key technologies, whose main function is to control the execute sequence of job and the distribution of computing resources, which directly relates to the Hadoop platform’s overall performance and system resources, usage. However, the existing job scheduling algorithms such as FIFO Scheduler, Fair Scheduler and Capacity Scheduler all have some defects. To overcome theses defects, this paper proposed a new algorithm Hadoop Greedy Scheduler Based on the Fair (HGSF). Firstly, the job pools are sorted by priority from high to low; pools in the same priority are sorted by their minimum requirements. Then their minimum requirements are met in turn. If have spare computing resources, it will be assigned to the job pool which has the highest priority and the minimum difference between the ideal requirement and the minimum requirement. Finally, the algorithm uses delay allocation strategy to improve the localization of the data for computing tasks. The experimental results testified the effectiveness of the proposed algorithm.