Comparison of MapReduce Systems

Project Information

Discipline
Computer Science (401) 
Subdiscipline
---401 Computer Science--- 
Orientation
Research 
Abstract

We are in a Big Data era. The rapid growth of information in science requires processing of large amounts of scientific data. One proposed solution is to apply data flow languages and runtimes to data intensive applications. The sample systems include the Google MapReduce, Microsoft Dryad, and CGL Twister. In this project, we will study applicability and performance of using those runtimes to solve Big Data issue in the science.

Intellectual Merit

Deploy runtimes such as Twister, Hadoop, Dryad on FutureGrid resources. Explore the applicability of new programming model and runtime technologies that can be used to solve thte Big Data issue in science.

Broader Impacts

Explore the feasibility of run data intensive applications with map reduce related technology on a dynamic,elastic, provisioned resources infrastructure. Abstract the design patterns of scientific applications for runtimes, such as Dryad, Twister, and Hadoop in HPC, Cluster, and Cloud.

Project Contact

Project Lead
Judy Qiu (xqiu) 
Project Manager
Yang Ruan (yangruan) 
Project Members
Yang Ruan, Yuduo Zhou, Judy Qiu  

Resource Requirements

Hardware Systems
  • hotel (IBM iDataPlex at U Chicago)
  • india (IBM iDataPlex at IU)
  • sierra (IBM iDataPlex at SDSC)
  • xray (Cray XM5 at IU)
  • bravo (large memory machine at IU)
 
Use of FutureGrid

Deploy runtimes such as Twister, Hadoop, Dryad on Future Grid resources.

Scale of Use

Samples of scale of use include: 1) Parallel SW-G job with 10,000 sequences on 32 nodes in Hadoop cluster. 2) Parallel Matrix Multiplication with the order of 31200 on 16 nodes in Dryad cluster. 3) Parallel Pagerank with 10GB web graph on 16 nodes in Twister cluster.

Project Timeline

Submitted
09/22/2010 - 06:36 
Completed
04/09/2013 
»