Running workflows in the cloud with Pegasus
Abstract
In this work we intend to study the benefits and drawbacks of using cloud computing for scientific workflows. In particular, we are interested in the benefits of specifying the execution environment of a workflow application as a virtual machine image. Using VM images has the potential to reduce the complexity of deploying workflow applications in distributed environments, and allow scientists to easily reproduce their experiments. In addition, we are interested in investigating the challenges of on-demand provisioning for scientific workflows in the cloud.
Intellectual Merit
Cloud computing is an important platform for future computational science applications. It is particularly well-suited for loosely-coupled applications such as scientific workflows, which do not require the high-speed interconnects and large, shared file systems typical of existing HPC systems. However, many of the current generation of workflow tools have been developed for the grid and may not be ready for use in the cloud. Although the cloud has many potential benefits, it also brings many additional challenges. We plan to investigate the use of clouds for workflows to determine what tools and techniques the workflow community will need to develop so that scientists using workflow technologies can take advantage of cloud computing.
Broader Impact
Many different science applications in physics, astronomy, molecular biology and earth science are using the Pegasus workflow management system in their research. These groups are interested in the potential benefits of cloud computing to improve the speed, quality, and reproducibility of their computational workloads. We intend to apply what we learn in using FutureGrid to develop tools and techniques to help scientists do their work better.
Use of FutureGrid
Running workflow applications in the cloud using Pegasus
Scale Of Use
I only need a few VMs. No more than 128 cores at a time.
Publications
- [Juve:2011:WVC:1996130.1996173] Juve, G., and E. Deelman, "Wrangler: virtual cluster provisioning for the cloud", ACM,
- [Vockler:2011:EUC:1996109.1996114] Vöckler, J. - S., G. Juve, E. Deelman, M. Rynge, and B. Berriman, "Experiences using cloud computing for a scientific workflow application", ACM,
- [6133211] Juve, G., and E. Deelman, "Automating Application Deployment in Infrastructure Clouds", 29 2011-dec. 1
Results
Jens-S. Vöckler, Gideon Juve, Ewa Deelman, Mats Rynge, G. Bruce Berriman, Experiences Using Cloud Computing for A Scientific Workflow Application, Proceedings of 2nd Workshop on Scientific Cloud Computing (ScienceCloud 2011), 2011.
Gideon Juve and Ewa Deelman, Wrangler: Virtual Cluster Provisioning for the Cloud, short paper, Proceedings of the 20th International Symposium on High Performance Distributed Computing (HPDC 2011), 2011.
Futuregrid is a resource provider for