Distributed and Parallel Programming Environments and their performance

Geoffrey Fox

This talk reviews and compares different programming models and their corresponding runtime for parallel and concurrent systems. There are a variety of hardware architectures varying from collections of cores on a chip, to clusters with customized interconnect and then widely distributed Grid systems. These vary in network bandwidth and latency as well as memory model (shared or distributed). Programming environments models exist at several levels including ambitious systems involving compilers and related tools decomposing applications for such systems. This is most important for multicore chips where approaches like OpenMP, PGAS, and the HPCS Languages could lead to programming breakthroughs for commodity systems. However much of recent progress has been in simpler concurrency models involving messaging to synchronize and coordinate nodes. Threads on cores can be coordinated with Microsoft CCR on a fine grain level and DSS at a coarser grain. Grids and Web services encouraged many different workflow approaches while MapReduce/Hadoop and Dryad developed a sophisticated dataflow approach familiar from earlier systems like AVS. Meanwhile MPI on cores and/or between nodes of a cluster or Grid remains a perfectly viable model although only popular in scientific computing.

Combining the different hardware with the different lower level programming models, we see a universal paradigm with execution units exchanging messages with other execution units with different tradeoffs in grain size, performance, fault tolerance and dynamic flexibility. 

We try to shed light on these old and emerging diverse hardware and software models with a comprehensive benchmarking activity built around "real" science applications. In this regard, we are developing scientifically interesting codes in C#, C++ and Java for several data mining applications. We use these codes to compare concurrency expressed on cores and/or nodes; the use of virtual machines as in some clouds; and the different programming models