Fortran With Message Passing

Message passing is a natural model of programming distributed memory MIMD computers, and is currently used in the vast majority of successful applications using MIMD machines. The basic idea is that each node (processor plus local memory) has a program which controls, and performs calculations on, its own data (the ``owner-computes'' rule). Non-local data may need to be obtained from other nodes, which is done by communication of messages.

In its simplest form, there is one program per node of the computer. The programs can be different (although usually they are the same), however they will generally follow different threads of control, for example different branches of an IF statement. Communication can be asynchronous, but in most cases the algorithms are loosely synchronous, [2] meaning that they are usually controlled by a time or iteration parameter and there is synchronization after every iteration, even though the communications during the iteration process may not be synchronous.

If parallelism is obtained from standard domain decomposition, then the parallel program for each node can look very similar to the sequential program, except that it computes only on local data, and has a call to a message passing routine to obtain non-local data. Schematically, a program might look something like the following:

2.0cm CALL COMMUNICATE (required non-local data)
2.0cm DO i running over local data
2.5cm CALL CALCULATE (with i's data)
2.0cm END DO

Note that it is more efficient to pass all the non-local data required in the loop as a single block before processing the local data, rather than pass each element of non-local data as it is needed within the loop. The advantages of this style of programming are:

• It is portable to both distributed and shared memory machines.
• It should scale to future machines, although to achieve good efficiencies schemes to overlap communication with itself and with calculation may be required.
• Languages are available now and are portable to many different MIMD machines. Current message passing language extensions include Express, PICL, PVM, and Linda.
• There will soon be an industry standard message passing interface. [16]
• All problems can be expressed using this method.

The disadvantages are:

• The user has complete control over transfer of data, which helps in creating efficient programs, but explicitly inserting all the communication calls is difficult, tedious, and error prone.
• Optimizations are not portable.
• It is only applicable to MIMD machines.