Full HTML for

Scripted foilset CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND

Given by Geoffrey C. Fox at Delivered Lectures of CPS615 Basic Simulation Track for Computational Science on 31 October 96. Foils prepared 11 November 1996
Outside Index Summary of Material Secs 74.8

This covers MPI from a user's point of view and is meant to be a supplement to other online resources from MPI Forum, David Walker's Tutorial, Ian Foster's "Designing and Building Parallel Programs", Gropp,Lusk and Skjellum "Using MPI"
An Overview is based on subset of 6 routines that cover send/receive, environment inquiry (for rank and total number of processors) initialize and finalization with simple examples
Processor Groups, Collective Communication and Computation and Derived Datatypes are also discussed

Table of Contents for full HTML of CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND

 Denote Foils where Image Critical
Denote Foils where HTML is sufficient
Indicates Available audio which is lightpurpleed out if missing
1 Delivered Lectures for CPS615 -- Base Course for the Simulation Track of Computational Science
Fall Semester 1996 --
Lecture of October 31 - 1996
2 Abstract of MPI -- The Message Passing Interface -- Presentation
3 MPI Overview -- Comparison with HPF -- I
4 MPI Overview -- Comparison with HPF -- II
5 Some Key Features of MPI
6 Some Difficulties with MPI
7 Why use Processor Groups?
8 MPI Conventions
9 Standard Constants in MPI
10 The Six Fundamental MPI routines
11 MPI_Init -- Environment Management
12 MPI_Comm_rank -- Environment Inquiry
13 MPI_Comm_size -- Environment Inquiry
14 MPI_Finalize -- Environment Management
15 Hello World in C plus MPI
16 Comments on Parallel Input/Output - I
17 Comments on Parallel Input/Output - II
18 Review of Message Passing Paradigm
19 Basic Point to Point Message Passing I
20 Basic Point to Point Message Passing II
21 Blocking Send MPI_Send(C) MPI_SEND(Fortran)
22 Blocking Receive MPI_Recv(C) MPI_RECV(Fortran)

Outside Index Summary of Material

HTML version of Scripted Foils prepared 11 November 1996

Foil 1 Delivered Lectures for CPS615 -- Base Course for the Simulation Track of Computational Science
Fall Semester 1996 --
Lecture of October 31 - 1996

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 21.6
Geoffrey Fox
NPAC
Room 3-131 CST
111 College Place
Syracuse NY 13244-4100
HTML version of Scripted Foils prepared 11 November 1996

Foil 2 Abstract of MPI -- The Message Passing Interface -- Presentation

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 74.8
This covers MPI from a user's point of view and is meant to be a supplement to other online resources from MPI Forum, David Walker's Tutorial, Ian Foster's "Designing and Building Parallel Programs", Gropp,Lusk and Skjellum "Using MPI"
An Overview is based on subset of 6 routines that cover send/receive, environment inquiry (for rank and total number of processors) initialize and finalization with simple examples
Processor Groups, Collective Communication and Computation and Derived Datatypes are also discussed
HTML version of Scripted Foils prepared 11 November 1996

Foil 3 MPI Overview -- Comparison with HPF -- I

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 87.8
MPI collected ideas from many previous message passing systems and put them into a "standard" so we could write portable (runs on all current machines) and scalable (runs on future machines we can think of) parallel software
MPI agreed May 1994 after a process that began with a workshop in April 1992
MPI plays same role to message passing systems that HPF does to data parallel languages
BUT whereas MPI has essentially all one could want -- as message passing fully understood
HPF will still evolve as many unsolved data parallel compiler issues
  • e.g. HPC++ -- the C++ version version of HPF still uncertain
  • and there is no data parallel version of C due to pointers (C* has restrictions)
  • HPJava is our new idea
  • whereas MPI fine with Fortran C or C++ and even Java
HTML version of Scripted Foils prepared 11 November 1996

Foil 4 MPI Overview -- Comparison with HPF -- II

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 105.1
HPF runs on SIMD and MIMD machines and is high level as it expresses a style of programming or problem architecture
MPI runs on MIMD machines (in principle it could run on SIMD but unnatural and inefficient) -- it expresses a machine architecture
Traditional Software Model is
  • Problem --> High Level Language --> Assembly Language --> Machine
  • -------- Expresses Problem -- Expresses Machine
So in this analogy MPI is universal "machine-language" of Parallel processing
HTML version of Scripted Foils prepared 11 November 1996

Foil 5 Some Key Features of MPI

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 185.7
Point to Point Message Passing
Collective Communication -- messages between >2 simultaneous processes
Support for Process Groups -- messaging in subsets of processors
Support for communication contexts -- general specification of message labels and ensuring unique to a set of routines as in a precompiled library
  • Note a communicator incorporates contexts and groups as single data structure and defines the scope of communication operation
Support for application (virtual) topologies analogous to distribution types in HPF
Inquiry routines to find out about environment such as number of processors
HTML version of Scripted Foils prepared 11 November 1996

Foil 6 Some Difficulties with MPI

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 407.5
Kitchen Sink has 129 functions and each has many arguments
  • This completeness is strength and weakness!
  • Hard to implement efficiently and hard to learn
It is not a complete operating environment and does not have ability to create and spawn processes etc.
PVM is previous dominant approach
  • It is very simple with much less functionality than MPI
  • However it runs on essentially all machines including workstation clusters
  • Further it is complete albeit simple operating environment
MPI outside distributed computing world with HTTP of the Web, ATM protocols and systems like ISIS from Cornell
However it does look as though MPI is being adopted as general messaging system by parallel computer vendors
HTML version of Scripted Foils prepared 11 November 1996

Foil 7 Why use Processor Groups?

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 231.8
We find a good example when we consider typical Matrix Algorithm
(matrix multiplication)
A i,j = Sk B i,k C k,j
summed over k'th column of B and k'th row of C
Consider a square decomposition of 16 by 16 matrices B and C as for Laplace's equation. (Good Choice)
Each operation involvea a subset(group) of 4 processors
HTML version of Scripted Foils prepared 11 November 1996

Foil 8 MPI Conventions

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 125.2
All MPI routines are prefixed by MPI_
  • C is always MPI_Xnnnnn(parameters) : C is case sensitive
  • Fortran is case insensitive but we will write MPI_XNNNNN(parameters)
MPI constants are in upper case as MPI datatype MPI_FLOAT for floating point number in C
Specify overall constants with
  • #include "mpi.h" in C programs
    • include "mpif.h" in Fortran
C routines are actually integer functions and always return a status (error) code
Fortran routines are really subroutines and have returned status code as argument
  • Please check on status codes although this is often skipped!
HTML version of Scripted Foils prepared 11 November 1996

Foil 9 Standard Constants in MPI

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 198.7
There a set of predefined constants in include files for each language and these include:
MPI_SUCCESS -- succesful return code
MPI_COMM_WORLD (everything) and MPI_COMM_SELF(current process) are predefined reserved communicators in C and Fortran
Fortran elementary datatypes are:
MPI_INTEGER, MPI_REAL, MPI_DOUBLE_PRECISION, MPI_COMPLEX, MPI_DOUBLE_COMPLEX, MPI_LOGICAL, MPI_CHARACTER, MPI_BYTE, MPI_PACKED
C elementary datatypes are:
MPI_CHAR, MPI_SHORT, MPI_INT, MPI_LONG, MPI_UNSIGNED_CHAR, MPI_UNSIGNED_SHORT, MPI_UNSIGNED, MPI_UNSIGNED_LONG, MPI_FLOAT, MPI_DOUBLE, MPI_LONG_DOUBLE, MPI_BYTE, MPI_PACKED
HTML version of Scripted Foils prepared 11 November 1996

Foil 10 The Six Fundamental MPI routines

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 243.3
call MPI_INIT(mpierr) -- initialize
call MPI_COMM_RANK (comm,rank,mpierr) -- find processor label (rank) in group
call MPI_COMM_SIZE(comm,size,mpierr) -- find total number of processors
call MPI_SEND (sndbuf,count,datatype,dest,tag,comm,mpierr) -- send a message
call MPI_RECV (recvbuf,count,datatype,source,tag,comm,status,mpierr) -- receive a message
call MPI_FINALIZE(mpierr) -- End Up
HTML version of Scripted Foils prepared 11 November 1996

Foil 11 MPI_Init -- Environment Management

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 72
This MUST be called to set up MPI before any other MPI routines may be called
For C: int MPI_Init(int *argc, char **argv[])
  • argc and argv[] are conventional C main routine arguments
  • As usual MPI_Init returns an error handle
For Fortran: call MPI_INIT(mpierr)
  • nonzero (more pedantically values not equal to MPI_SUCCESS) values of mpierr represent errors
HTML version of Scripted Foils prepared 11 November 1996

Foil 12 MPI_Comm_rank -- Environment Inquiry

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 136.8
This allows you to identify each processor by a unique integer called the rank which runs from 0 to N-1 where there are N processors
If we divide the region 0 to 1 by domain decomposition into N parts, the processor with rank r controls
  • region: r/N to (r+1)/N
for C:int MPI_Comm_rank(MPI_Comm comm, int *rank)
  • comm is an MPI communicator of type MPI_Comm
for FORTRAN: call MPI_COMM_RANK (comm,rank,mpierr)
HTML version of Scripted Foils prepared 11 November 1996

Foil 13 MPI_Comm_size -- Environment Inquiry

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 24.4
This returns in integer size number of processes in given communicator comm (remember this specifies processor group)
For C: int MPI_Comm_size(MPI_Comm comm,int *size)
For Fortran: call MPI_COMM_SIZE (comm,size,mpierr)
  • where comm, size, mpierr are integers
  • comm is input; size mpierr returned
HTML version of Scripted Foils prepared 11 November 1996

Foil 14 MPI_Finalize -- Environment Management

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 36
Before exiting, an MPI application it is courteous to clean up the MPI state and MPI_FINALIZE does this. No MPI routine may be called in a given process after that process has called MPI_FINALIZE
for C: int MPI_Finalize()
for Fortran: MPI_FINALIZE(mpierr)
  • mpierr is an integer
HTML version of Scripted Foils prepared 11 November 1996

Foil 15 Hello World in C plus MPI

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 115.2
#include <stdio.h>
#include <mpi.h>
void main(int argc,char *argv[])
{
  • int ierror, rank, size
  • MPI_Init(&argc, &argv); # Initialize
  • MPI_Comm_rank(MPI_COMM_WORLD, &rank); # Find Processor Number
  • if( rank == 0)
    • printf ("hello World!\n");
  • ierror=MPI_Comm_size(MPI_COMM_WORLD, &size);
  • if( ierror != MPI_SUCCESS ) # Find Total number of processors above
    • MPI_Abort(MPI_COMM_WORLD,ierror); # Abort
  • printf("I am processor %d out of total of %d\n", rank, size);
  • MPI_Finalize(); # Finalize
}
HTML version of Scripted Foils prepared 11 November 1996

Foil 16 Comments on Parallel Input/Output - I

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 306.7
Parallel I/O has technical issues -- how best to optimize access to a file whose contents may be stored on N different disks which can deliver data in parallel and
Semantic issues -- what does printf in C (and PRINT in Fortran) mean?
The meaning of printf/PRINT is both undefined and changing
  • In my old Caltech days, printf on a node of a parallel machine was a modification of UNIX which automatically transferred data from nodes to "host" and produced a single stream
  • In those days, full UNIX did not run on every node of machine
  • We introduced new UNIX I/O modes (singular and multiple) to define meaning of parallel I/O and I thought this was a great idea but it didn't catch on!!
HTML version of Scripted Foils prepared 11 November 1996

Foil 17 Comments on Parallel Input/Output - II

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 148.3
Today, memory costs have declined and ALL mainstream MIMD distributed memory machines whether clusters of workstations or integrated systems such as T3D/ Paragon/ SP-2 have enough memory on each node to run UNIX
Thus printf today means typically that the node on which it runs will stick it out on "standard output" file for that node
  • However this is implementation dependent
  • e.g. If you want a stream of output with information in order
    • Say that from node 0, then node 1, then node 2 etc.
    • This was default on old Caltech machines but
  • In general you need to communicate information from nodes 1 to N-1 to node 0 and let node 0 sort it and output in required order
New MPI-IO initiative will link I/O to MPI in a standard fashion
HTML version of Scripted Foils prepared 11 November 1996

Foil 18 Review of Message Passing Paradigm

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 649.4
Data is propagated between processors via messages which can be divided into packets but at MPI level we only see logically single complete messages
The building block is Point to Point Communication with one processor sending information and one other receiving it
Collective communication involves more than one message
  • Broadcast of one processor to group of other processors (call a multicast on Internet when restricted to group)
  • Synchronization or barrier
  • Exchange of Information between processors
  • Reduction Operation such as a Global Sum
Collective Communication can ALWAYS be implemented in terms of elementary point to point communications but is provided
  • for user convenience and
  • often the best algorithm is hardware dependent and "official" MPI collective routines ought to be faster than portable implementation in terms of point to point primitive routines.
HTML version of Scripted Foils prepared 11 November 1996

Foil 19 Basic Point to Point Message Passing I

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 146.8
Communication is between two processors and receiving process must expect a message although can be uncertain asto message type and sending process
Information required to specify a message includes
  • Identification of sender process(or)
  • Identification of destination process
  • Type of data -- Integers, Floats, Characaters -- note floating point numbers may need conversion if sending between machines of different type
  • Number of data elements to send and in destination process, maximum size of message that can be received
  • Where the data lives in sending process -- typically a pointer
  • Where the received data should be stored into -- again a pointer
HTML version of Scripted Foils prepared 11 November 1996

Foil 20 Basic Point to Point Message Passing II

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 685.4
Two types of communication operations applicable to send and receive
  • Blocking -- sender and receiver wait until communication is complete
  • Non-blocking -- send and receive handled concurrently with computation -- MPI send and receive routines return immediately before message processed so processor can do other things
In addition four types of send operation
  • Standard -- A send may be initiated even if matching receive has not been initiated
  • Synchronous -- Waits for recipent to complete and so message delivery guaranteed
  • Buffered -- Copies message into a buffer (if necessary) on destination processor so that completion of send independent of matching receive
  • Ready -- A send is only started when matching receive initiated
HTML version of Scripted Foils prepared 11 November 1996

Foil 21 Blocking Send MPI_Send(C) MPI_SEND(Fortran)

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 260.6
call MPI_SEND (
  • IN message start address of data to send
  • IN message_len number of items (length in bytes
    • determined by type)
  • IN datatype type of each data element
  • IN dest_rank Processor number (rank) of destination
  • IN message_tag tag of message to allow receiver to filter
  • IN communicator Communicator of both sender and receiver group
  • OUT error_message) Error Flag (absent in C)
Fortran example:
  • integer count, datatype, dest, tag, comm, mpierr
  • comm= MPI_COMM_WORLD
  • tag=0
  • datatype=MPI_REAL
call MPI_SEND (sndbuf,count,datatype,dest,tag,comm,mpierr)
HTML version of Scripted Foils prepared 11 November 1996

Foil 22 Blocking Receive MPI_Recv(C) MPI_RECV(Fortran)

From CPS615-Initial Lecture on MPI ending with discussion of basic MPI_SEND Delivered Lectures of CPS615 Basic Simulation Track for Computational Science -- 31 October 96. *
Full HTML Index Secs 279.3
call MPI_RECV(
  • IN start_of_buffer Address of place to store data(address is INput
    • values of data are of course output starting at this adddress!)
  • IN buffer_len Maximum number of items allowed
  • IN datatype Type of each data type
  • IN source_rank Processor number (rank) of source
  • IN tag only accept messages with this tag value
  • IN communicator Communicator of both sender and receiver group
  • OUT return_status Data structure describing what happened!
  • OUT error_message) Error Flag (absent in C)
Note that return_status is used after completion of receive to find actual received length (buffer_len is a MAXIMUM), actual source processor rank and actual message tag
In C syntax is
int error_message=MPI_Recv(void *start_of_buffer,int buffer_len, MPI_DATATYPE datatype, int source_rank, int tag, MPI_Comm communicator, MPI_Status *return_status)
© Northeast Parallel Architectures Center, Syracuse University, npac@npac.syr.edu

If you have any comments about this server, send e-mail to webmaster@npac.syr.edu.

Page produced by wwwfoil on Fri Aug 15 1997