FRIEDA: Flexible Robust Intelligent Elastic Data Management
Abstract
Scientific applications are increasingly using cloud resources for their data analysis workflows. We use the cloud loosely to signify transient environments. However, managing data effectively and efficiently over these cloud resources is challenging due to the myriad storage choices with different performance, cost trade-offs, complex application choices and complexity associated with elasticity, failure rates in these environments. The different data access patterns for data-intensive scientific applications require a more flexible and robust data management solution than the ones currently in existence. FRIEDA is a Flexible Robust Intelligent Elastic Data Management framework that employs a range of data management strategies approaches in elastic environments.
Specifically, we are investigating
- Semi-automated storage choices and data management strategies for data analysis science workflows
- Management of the life cycle of scientific applications in hybrid environments using HPC and cloud resources.
Intellectual Merit
The proposed research will significantly impact current understanding on storage and data management in CS. The approaches will be useful for data-intensive applications in both cloud and future HPC environments
Broader Impact
The project is expected to impact scientific discoveries through use of cloud resources.
Use of FutureGrid
FutureGrid will be used to investigate the following issues -
* trade-offs between different storage options in cloud environments
* elastic data management at scale in cloud environments
Scale Of Use
Many of the things we need to test will need to be done at scale since a lot of the data management issues are at scale. We will start with a few nodes used for development, testing and will then need to be able to scale up for tests. We will take as much as you can give us over the sites.
Futuregrid is a resource provider for