Table of Contents

• Barrier synchronization
• Broadcast from one member to all other members
• Gather data from an array spread across processors into one array
• Scatter data from one member to all members
• All-to-all exchange of data
• Global reduction (e.g., sum, min of "common" data elements)
• Scan across all members of a communicator

C

FORTRAN

where:

buffer	is the starting address of a buffer,
count	is an integer indicating the number of data elements in the buffer,
datatype	is MPI defined constant indicating the data type of the elements in the buffer,
root	is an integer indicating the rank of broadcast root process, and
comm	is the communicator.

The MPI_BCAST must be called by each node in the group, specifying the same comm and root. The message is sent from the root process to all processes in the group, including the root process.

MPI_GATHERV and MPI_SCATTERV are the varying versions of MPI_GATHER and MPI_SCATTER. MPI_GATHERV extends the functionality of MPI_GATHER by changing the receive count from an integer to an integer array and providing a new argument displs(array). MPI_GATHERV allows a varying count of data from each processor and allows some flexibility for where the gathered data is placed in the root as well. As a counterpart of MPI_GATHERV, MPI_SCATTERV is an extension of MPI_SCATTER in the same relationship as MPI_GATHERV is to MPI_GATHER.

C

FORTRAN

The variables for Gatherv are:

sbuf	starting address of send buffer,
scount	number of elements in send buffer,
stype	data type of send buffer elements,
rbuf	starting address of receive buffer,
rcount	array containing number of elements to be received from each process,
displs	array specifying the displacement relative to rbuf at which to place the incoming data from corresponding process,
rtype	data type of receive buffer,
root	rank of receiving process,
comm	group communicator.

Note: rbuf, rcount, rtype are significant for the root process only.

The variables for Scatterv are:

sbuf	address of send buffer,
scount	integer array specifying the number of elements to send to each process,
displs	array specifying the displacement relative to sbuf from which to take the data going out to the corresponding process,
stype	data type of send buffer elements,
rbuf	address of receive buffer,
rcount	number of elements in receive buffer,
rtype	data type of receive buffer elements,
root	rank of sending process,
comm	group communicator

Note: sbuf significant for root process only

For the purpose of illustrating the usage of MPI_GATHERV and MPI_SCATTERV, we give two Fortran program fragments below:

MPI_GATHERV

MPI_SCATTERV

MPI_ALLGATHER can be thought of as MPI_GATHER where all processes, not just the root, receive the result. The jth block of the receive buffer is the block of data sent from the jth process. A similar relationship holds for MPI_ALLGATHERV and MPI_GATHERV. The syntax of MPI_ALLGATHER and MPI_ALLGATHERV are similar to MPI_GATHER and MPI_GATHERV, respectively. However, the argument root is dropped from MPI_ALLGATHER and MPI_ALLGATHERV.

C

FORTRAN

The variables for Allgather are:

sbuf	starting address of send buffer,
scount	number of elements in send buffer,
stype	data type of send buffer elements,
rbuf	address of receive buffer,
rcount	number of elements received from any process,
rtype	data type of receive buffer elements,
comm	group communicator.

Note: the arguments are the same as MPI_GATHER or MPI_GATHERV except for no root argument.

Example:

• Matrix-vector multiplication of matrix distributed by rows
• Output vector needed in entirety by all processors

Sample Code

      DIMENSION A(25,100), b(100), cpart(25), ctotal(100)

      DO I=1,25
            cpart(I) = 0.
            DO K=1,100
                  cpart(I) = cpart(I) + A(I,K)*b(K)
            END DO
      END DO
      call MPI_ALLGATHER(cpart,25,MPI_REAL,ctotal,25,
            MPI_REAL, MPI_COMM_WORLD, ierr)

MPI_REDUCE

One of the most useful collective operations is a global reduction or combine operation. The partial result in each process in the group is combined in one specified process or all the processes using some desired function. If there are n processes in the process group, and D(i,j) is the jth data item in process i, then the Dj item of data in the root process resulting from a reduce routine evaluation is given by:

where * is the reduction function and it is always assumed associative. All MPI predefined functions are also assumed to be commutative. One may define functions that are assumed to be associative, but not commutative. Each process can provide either one element or a sequence of elements. In both cases the combine operation is executed element-wise on each element of the sequence. There are three versions of reduce. They are MPI_REDUCE, MPI_ALLREDUCE, and MPI_REDUCE_SCATTER. The form of these reduction primitives is listed below:

C

FORTRAN

The differences among these three reduces:

• MPI_REDUCE returns results to a single process;
• MPI_ALLREDUCE returns results to all processes in the group;
• MPI_REDUCE_SCATTER scatters a vector, which results from a reduce operation, across all processes.

Notes:

* rbuf significant only at the root process,

* the argument rcounts in MPI_REDUCE_SCATTER is an array.

Scan

where * is the reduction function which may be either an MPI predefined function or a user defined function.

The syntax of the MPI scan routine is

C

FORTRAN

Predefined Reduce Operations

Examples of MPI predefined operations are summarized in Table 1. MPI also provides a mechanism for user-defined operations used in MPI_ALLREDUCE and MPI_REDUCE.

Example

• Each process in a forest dynamics simulation calculates the maximum tree height for its region.
• Processor 0, which is writing output, must know the global maximum height.

  INTEGER maxht, globmx
      .
      .
      .  (calculations which determine maximum height)
      .
      .
  call MPI_REDUCE (maxht, globmx, 1, MPI_INTEGER, MPI_MAX, 0, MPI_COMM_WORLD, ierr)
  IF (taskid.eq.0) then
      .
      .  (Write output)
      .
  END IF

User-defined Operations

• User can define his/her own reduce operation
• Makes use of the MPI_OP_CREATE function

3. Performance Issues

• A great deal of hidden communication takes place with collective communication
• Performance depends greatly on the particular implementation of MPI
• Because there may be forced synchronization, not always best to use collective communication

3.1 Example: Broadcast to 8 Processors

Example of two ways (simple way, better way) to do a broadcast.

Simple approach: One processor broadcasts the same message to all the other processors, one at a time.

Amount of data transferred: (N-1)*p

N = number of processors
p = size of message
Number of steps: N-1

Compare the number of steps in the two approaches:

Simple approach: (N-1)*p
Smarter approach: log₂N

The latter scales better as N increases.

Example of two ways (simple way, better way) to do a scatter.

Simple approach: One processor scatters the N messages to the other processors, one at a time.

Amount of data transferred: (N-1)*p

N = number of processors
p = size of message
Number of steps: (N-1)*p

Compare the number of steps in the two approaches:

Simple approach: (N-1)*p
Smarter approach: log₂N

The latter scales better as N increases.