Network Aware Task Scheduling in Hadoop

Hadoop, an open-source MapReduce platform, has been widely used for big data processing in large scale distributed systems. Based on scalable cloud computing deployed in data centers, Hadoop runs on a virtualized cluster and its performance is closely tied to its task scheduler, which implicitly assumes that cluster nodes are homogenous. However, performance variation (referred to as jitter) exists among the virtualized nodes and impacts overall performance of workloads running on the Hadoop. The sources of variation come from a number of places such as: network congestion, disk contention, CPU overload, running out of memory, system failures and crashes, and etc. The scheduler has to reinvestigate the above problems and take more metrics into consideration when dispatching tasks to individual nodes where task tracker and data storage are both running. The conventional scheduling principle is to place a computing task as close to the data as possible. If a task is assigned to a node that does not contain the necessary data, the node will have to fetch data from a remote node sacrificing performance and wasting energy. Several existing research projects have pointed that network topology affects Hadoop performance and there are overloaded or congested paths in Hadoop network when running intensive workload. In our initial experiments, we found some interesting results from collected traces by running intensive workload like hive benchmark, pigmix, nutch, and TPC-H. A large number of Map tasks were scheduled remotely where network traffic would be an issue. Thus, we plan to design a new data-locality and network aware scheduling algorithm for Hadoop to address the above issues in terms of network traffic. The proposed scheduling algorithm not only follows data-locality principle but also utilizes cluster network traffic information to direct task assignment. We already designed and implemented a Hadoop simulator named HadoopSim integrated with a network simulator NS-3. HadoopSim replays the traces and simulates all Hadoop events and data transfers over network. We will design a new data-locality aware task scheduler in HadoopSim and quantify the new scheduler in terms of performance and energy efficiency compared with several existing schedulers like: FIFO scheduler, capacity scheduler, and fair scheduler.