LsPS: A Job Size-Based Scheduler for Efficient Task Assignments in Hadoop

The MapReduce paradigm and its open source implementation Hadoop are emerging as an important standard for large-scale data-intensive processing in both industry and academia. A MapReduce cluster is typically shared among multiple users with different types of workloads. When a flock of jobs are concurrently submitted to a MapReduce cluster, they compete for the shared resources and the overall system performance in terms of job response times, might be seriously degraded. Therefore, one challenging issue is the ability of efficient scheduling in such a shared MapReduce environment. However, we find that conventional scheduling algorithms supported by Hadoop cannot always guarantee good average response times under different workloads. To address this issue, we propose a new Hadoop scheduler, which leverages the knowledge of workload patterns to reduce average job response times by dynamically tuning the resource shares among users and the scheduling algorithms for each user. Both simulation and real experimental results from Amazon EC2 cluster show that our scheduler reduces the average MapReduce job response time under a variety of system workloads compared to the existing FIFO and Fair schedulers.