banner

Batch Parallel Programming

This section covers the integration of Hadoop into the Pilot Jobs framework (link) more details of which can be found in the Software Tools page, and the Harp scientific (high performance) plug-in for Hadoop which benchmarks GML5 (link).

Streaming Data

The following diagram illustrates the Apache Storm method of processing data from the Internet of Things (IoT).
Big Data Stack HPC-ABDS
  • Research paper here.
  • Research paper here.
  • Research paper here.
  • Research paper here.

MIDAS Ongoing Work

  1. Apply Storm-RabbitMQ to Robotics
  2. Improve Storm to add quality of service and support of parallel computing
  3. Extend Tweet clustering to use Harp
  4. Look at further examples of Harp
  5. Integrate Pilot Jobs with Streaming and Harp
  6. Support GPU and Xeon Phi