Technology Used
We will use FutureSystems (previously FutureGrid) facilities and cloud computing experience is helpful but not essential. Good working experience with Java is required.
Instructor
Geoffrey Charles Fox
Senior Associate Dean for Research
Distinguished Professor of Physics,
Computer Science and Informatics
gcf@indiana.edu
Course Information
The course covers the following material
This course studies software used in many commercial activities to study Big Data. The backdrop for course is the ~300 software subsystems illustrated at http://hpc-abds.org/kaleidoscope/ (Links to an external site.). We will describe the software architecture represented by this collection which we term HPC-ABDS (High Performance Computing - enhanced Apache Big Data Stack).
- The cloud computing architecture underlying ABDS and contrast of this with HPC.
- The software architecture with its different layers at HPC-ABDS Kaleidoscope covering broad functionality and rationale for each layer.
- Then we will go through selected software systems – about 5% of those in the Kaleidoscope which have been already deployed on FutureSystems (previously called FutureGrid) cloud using OpenStack and Chef recipes.
- Then we will go through selected software systems – about 10% of those in the Kaleidoscope which have been already deployed on FutureSystems systems using OpenStack and Chef recipes.
- Students will chose one other open source member of Kaleidoscope each and deploy as illustrated in class.
- The main activity of the course will be building a significant project using multiple HPC-ABDS subsystems combined with user code and data.
- Projects will be suggested or students can chose their own.
- Elementary knowledge in a scripting language needed (if not available this can be acquired as part of this course)
- Basic knowledge of Python desirable (if not available this can be acquired as part of this course)
- Ability to (learn to) use the LInux/Unix command shell (we will have a lesson on this)
- Basic understanding on how to install packages and programs on Linux (we will have a lesson on this)
- DevOps: "software deployment automation".
- Linux command shell.
- Elementary usage of ssh.
- Use of Github to store software packages and documentation.
- The reproducible installation of sophisticated platforms on virtual clusters. This is facilitated either by scripts developed in Python, Openstack Heat, or a DevOps framework such as Ansible, Chef, or Puppet. Which framework is chosen will depend on the experience level of the student.
- You will learn utility of the key parts of Big Data Stack
This course will be incremented with new material. A full set of course material from previous Fall 2014 offering of the class is accessible here.
Course Lessons
- Part A: Introduction 13:59
- Part B: Real World Big Data 15:28
- Part C: Basic Trends and Jobs 12:24
Time
- Part A: Introduction to HPC-ABDS Software and Access Patterns 27:45
- Part B: Science Examples (Data Access Patterns) 18:38
- Part C: Remaining General Access Patterns 11:26
- Part D: Summary of HPC-ABDS Layers 1 - 6 14:32
- Part E: Summary of HPC-ABDS Layers 7 - 13 30:52
- Part F: Summary of HPC-ABDS Layers 14 - 17 28:02
- Part G: Final Part Summary of Stack 20:20
Time
- Part A: More of Software Stack 24:00
Time
In this week, you will learn how to gain access to the FutureSystems resources. Some of the lessons have been prepared for the beginners to help understand the basics of Linux operating systems and the collaboration tools i.e. github, google hangout and remote desktop. Please watch video lessons and read online materials on this page. It also covers Unix shell scripting, SSH and other utilities with various exercises.
In this week, you will learn about OpenStack ad Public Clouds. OpenStack is a open-source cloud computing software platform and a community-driven project. You can use OpenStack to build a cloud infrastructure in your public or private network, or you can simply use cloud software for your services. The lessons in this week are specifically prepared to try OpenStack Software and give you the confidence and understanding of using IaaS cloud platforms. There are tutorial lessons to explore OpenStack web dashboard (Horizon) and compute engine (Nova) including Public Clouds e.g. Amazon EC2 or Microsoft Azure.
In this week, you will learn about Cloudmesh which is a cloud resource management software written in Python. It automates launching multiple VM instances across different cloud platforms including Amazon EC2, Microsoft Azure Virtual Machine, HP Cloud, OpenStack, and Eucalyptus. The web interface of Cloudmesh help users and administrators manage entire cloud resources. Most cutting-edge technologies such as Apache LibCloud, Celery, IPython, Flask, Fabric, Docopt, YAML, MongoDB, and Sphinx are applied to enhance Web Service, Command Line Tools and Rest APIs.
In this week, you will learn about open-source configuration management (CM) software as part of IT automation and orchestration. We focus on Ansible and OpenStack Heat to review of system configuration and management but Salt, Puppet, Chef, and Juju are introduced to explore other tools as well. With different features of these software, you will see which tool is ideal for your system environment and understand basic CM techniques. We have a few lab sessions to provide hands-on experience about deploying and configuring applications on IT infrastructure.
This week, you will learn basics of virtual clusters. Typically, analyzing large data sets containing unstructured data types requires distributed computing resources for data processing with high performance, scalability, and availability. With virtualization technology, cluster computing can be more flexible, effective and cost-efficient in terms of resource utilization. There are three basic tutorials about deploying a virtual cluster, Hadoop cluster and MongoDB Sharded cluster which give you a chance to gain some experience of how to setup virtual clusters manually and configure software with Cloudmesh.
Unit 8: Overview of HPC-ABDS Software Stack
This consists of the ~300 technologies in HPC-ABDS Described with roughly one page per technology. It is divided into several lessons in separate PowerPoints organized by layers of HPC-ABDS technology.
- Layers 1-2: Message Protocols and Distributed Coordination PPT 19:18
- Layers 3-4: Security & Privacy and Monitoring PPT 20:51
- Layer 5: IaaS Management from HPC to Hypervisors Part 1 PPT 19:00
- Layer 5: IaaS Management from HPC to Hypervisors Part 2 PPT 18:29
- Layer 6: DevOps Part 1 PPT 24:18
- Layer 6: DevOps Part 2 PPT 22:45
- Layer 7: Interoperability PPT 15:35
- Layer 8: File Systems PPT 29:33
- Layer 9: Cluster Resource Management PPT 28:27
- Layer 10: Data Transport PPT 17:17
- Layer 11A: File Management PPT 20:14
- Layer 11B: NoSQL Part 1 PPT 25:03
- Layer 11B: NoSQL Part 2 PPT 22:57
- Layer 11C: SQL PPT 25:31
- Layer 12: In-Memory Databases & Caches; Object-Relational Mapping; Extraction Tools PPT 23:02
- Layer 13: Inter Process Communication Collectives, Point to Point, Publish-Subscribe, MPI Part 1 PPT 20:43
- Layer 13: Inter Process Communication Collectives, Point to Point, Publish-Subscribe, MPI Part 2 PPT 11:17
- Layer 14A: Basic Programming Model and Runtime, SPMD, MapReduce PPT 20:50
- Layer 14B: Streaming PPT 17:43
- Layer 15A: High Level Programming PPT 28:02
- Layer 15B: Application Hosting Frameworks Part 1 PPT 14:38
- Layer 15B: Application Hosting Frameworks Part 2 PPT 13:23
- Layer 16: Applications and Analytics Part 1 PPT 22:52
- Layer 16: Applications and Analytics Part 2 PPT 25:39
- Layer 17: Workflow Orchestration Part 1) Pre-Cloud PPT 24:25
- Layer 17: Workflow Orchestration Part 2) Cloud PPT 31:37
Title
Slides
Time
FutureSystems Access
- Create an account on the FutureSystems Portal.
- Request to be added to project FG-465.
- Upload a public SSH key to FutureSystems portal in order to access FutureSystems systems. Initial steps described in videos 1) Get a Portal Account, and 2) Upload an SSH key
- Explore the OpenStack Tutorial.
- Instructions for account creation, joining a project and uploading an SSH key are all available here.
- If you are using Windows, the simplest solution for using SSH keys is the Putty SSH client, and it’s SSH authentication agent Pageant. Putty and its associated programs are available here
