Fox Presentation Fall 1995 The Message Passing Interface MPI for CPS 615 -- Computational Science in Simulation Track October 1, 1995 Updated Oct 31 1996 Geoffrey Fox NPAC Syracuse University 111 College Place Syracuse NY 13244-4100 Abstract of MPI -- The Message Passing Interface -- Presentation This covers MPI from a user's point of view and is meant to be a supplement to other online resources from MPI Forum, David Walker's Tutorial, Ian Foster's "Designing and Building Parallel Programs", Gropp,Lusk and Skjellum "Using MPI" An Overview is based on subset of 6 routines that cover send/receive, environment inquiry (for rank and total number of processors) initialize and finalization with simple examples Processor Groups, Collective Communication and Computation and Derived Datatypes are also discussed MPI Overview -- Comparison with HPF -- I MPI collected ideas from many previous message passing systems and put them into a "standard" so we could write portable (runs on all current machines) and scalable (runs on future machines we can think of) parallel software MPI agreed May 1994 after a process that began with a workshop in April 1992 MPI plays same role to message passing systems that HPF does to data parallel languages BUT whereas MPI has essentially all one could want -- as message passing fully understood HPF will still evolve as many unsolved data parallel compiler issues e.g. HPC++ -- the C++ version version of HPF still uncertain and there is no data parallel version of C due to pointers (C* has restrictions) HPJava is our new idea whereas MPI fine with Fortran C or C++ and even Java MPI Overview -- Comparison with HPF -- II HPF runs on SIMD and MIMD machines and is high level as it expresses a style of programming or problem architecture MPI runs on MIMD machines (in principle it could run on SIMD but unnatural and inefficient) -- it expresses a machine architecture Traditional Software Model is Problem --> High Level Language --> Assembly Language --> Machine -------- Expresses Problem -- Expresses Machine So in this analogy MPI is universal "machine-language" of Parallel processing Some Key Features of MPI Point to Point Message Passing Collective Communication -- messages between >2 simultaneous processes Support for Process Groups -- messaging in subsets of processors Support for communication contexts -- general specification of message labels and ensuring unique to a set of routines as in a precompiled library Note a communicator incorporates contexts and groups as single data structure and defines the scope of communication operation Support for application (virtual) topologies analogous to distribution types in HPF Inquiry routines to find out about environment such as number of processors Some Difficulties with MPI Kitchen Sink has 129 functions and each has many arguments This completeness is strength and weakness! Hard to implement efficiently and hard to learn It is not a complete operating environment and does not have ability to create and spawn processes etc. PVM is previous dominant approach It is very simple with much less functionality than MPI However it runs on essentially all machines including workstation clusters Further it is complete albeit simple operating environment MPI outside distributed computing world with HTTP of the Web, ATM protocols and systems like ISIS from Cornell However it does look as though MPI is being adopted as general messaging system by parallel computer vendors Why use Processor Groups? We find a good example when we consider typical Matrix Algorithm (matrix multiplication) A i,j = Sk B i,k C k,j summed over k'th column of B and k'th row of C Consider a square decomposition of 16 by 16 matrices B and C as for Laplace's equation. (Good Choice) Each operation involvea a subset(group) of 4 processors MPI Conventions All MPI routines are prefixed by MPI_ C is always MPI_Xnnnnn(parameters) : C is case sensitive Fortran is case insensitive but we will write MPI_XNNNNN(parameters) MPI constants are in upper case as MPI datatype MPI_FLOAT for floating point number in C Specify overall constants with #include "mpi.h" in C programs include "mpif.h" in Fortran C routines are actually integer functions and always return a status (error) code Fortran routines are really subroutines and have returned status code as argument Please check on status codes although this is often skipped! Standard Constants in MPI There a set of predefined constants in include files for each language and these include: MPI_SUCCESS -- succesful return code MPI_COMM_WORLD (everything) and MPI_COMM_SELF(current process) are predefined reserved communicators in C and Fortran Fortran elementary datatypes are: MPI_INTEGER, MPI_REAL, MPI_DOUBLE_PRECISION, MPI_COMPLEX, MPI_DOUBLE_COMPLEX, MPI_LOGICAL, MPI_CHARACTER, MPI_BYTE, MPI_PACKED C elementary datatypes are: MPI_CHAR, MPI_SHORT, MPI_INT, MPI_LONG, MPI_UNSIGNED_CHAR, MPI_UNSIGNED_SHORT, MPI_UNSIGNED, MPI_UNSIGNED_LONG, MPI_FLOAT, MPI_DOUBLE, MPI_LONG_DOUBLE, MPI_BYTE, MPI_PACKED The Six Fundamental MPI routines call MPI_INIT(mpierr) -- initialize call MPI_COMM_RANK (comm,rank,mpierr) -- find processor label (rank) in group call MPI_COMM_SIZE(comm,size,mpierr) -- find total number of processors call MPI_SEND (sndbuf,count,datatype,dest,tag,comm,mpierr) -- send a message call MPI_RECV (recvbuf,count,datatype,source,tag,comm,status,mpierr) -- receive a message call MPI_FINALIZE(mpierr) -- End Up MPI_Init -- Environment Management This MUST be called to set up MPI before any other MPI routines may be called For C: int MPI_Init(int *argc, char **argv[]) argc and argv[] are conventional C main routine arguments As usual MPI_Init returns an error handle For Fortran: call MPI_INIT(mpierr) nonzero (more pedantically values not equal to MPI_SUCCESS) values of mpierr represent errors MPI_Comm_rank -- Environment Inquiry This allows you to identify each processor by a unique integer called the rank which runs from 0 to N-1 where there are N processors If we divide the region 0 to 1 by domain decomposition into N parts, the processor with rank r controls region: r/N to (r+1)/N for C:int MPI_Comm_rank(MPI_Comm comm, int *rank) comm is an MPI communicator of type MPI_Comm for FORTRAN: call MPI_COMM_RANK (comm,rank,mpierr) MPI_Comm_size -- Environment Inquiry This returns in integer size number of processes in given communicator comm (remember this specifies processor group) For C: int MPI_Comm_size(MPI_Comm comm,int *size) For Fortran: call MPI_COMM_SIZE (comm,size,mpierr) where comm, size, mpierr are integers comm is input; size mpierr returned MPI_Finalize -- Environment Management Before exiting, an MPI application it is courteous to clean up the MPI state and MPI_FINALIZE does this. No MPI routine may be called in a given process after that process has called MPI_FINALIZE for C: int MPI_Finalize() for Fortran: MPI_FINALIZE(mpierr) mpierr is an integer Hello World in C plus MPI #include #include void main(int argc,char *argv[]) { int ierror, rank, size MPI_Init(&argc, &argv); # Initialize MPI_Comm_rank(MPI_COMM_WORLD, &rank); # Find Processor Number if( rank == 0) printf ("hello World!\n"); ierror=MPI_Comm_size(MPI_COMM_WORLD, &size); if( ierror != MPI_SUCCESS ) # Find Total number of processors above MPI_Abort(MPI_COMM_WORLD,ierror); # Abort printf("I am processor %d out of total of %d\n", rank, size); MPI_Finalize(); # Finalize } Comments on Parallel Input/Output - I Parallel I/O has technical issues -- how best to optimize access to a file whose contents may be stored on N different disks which can deliver data in parallel and Semantic issues -- what does printf in C (and PRINT in Fortran) mean? The meaning of printf/PRINT is both undefined and changing In my old Caltech days, printf on a node of a parallel machine was a modification of UNIX which automatically transferred data from nodes to "host" and produced a single stream In those days, full UNIX did not run on every node of machine We introduced new UNIX I/O modes (singular and multiple) to define meaning of parallel I/O and I thought this was a great idea but it didn't catch on!! Comments on Parallel Input/Output - II Today, memory costs have declined and ALL mainstream MIMD distributed memory machines whether clusters of workstations or integrated systems such as T3D/ Paragon/ SP-2 have enough memory on each node to run UNIX Thus printf today means typically that the node on which it runs will stick it out on "standard output" file for that node However this is implementation dependent e.g. If you want a stream of output with information in order Say that from node 0, then node 1, then node 2 etc. This was default on old Caltech machines but In general you need to communicate information from nodes 1 to N-1 to node 0 and let node 0 sort it and output in required order New MPI-IO initiative will link I/O to MPI in a standard fashion Review of Message Passing Paradigm Data is propagated between processors via messages which can be divided into packets but at MPI level we only see logically single complete messages The building block is Point to Point Communication with one processor sending information and one other receiving it Collective communication involves more than one message Broadcast of one processor to group of other processors (call a multicast on Internet when restricted to group) Synchronization or barrier Exchange of Information between processors Reduction Operation such as a Global Sum Collective Communication can ALWAYS be implemented in terms of elementary point to point communications but is provided for user convenience and often the best algorithm is hardware dependent and "official" MPI collective routines ought to be faster than portable implementation in terms of point to point primitive routines. Basic Point to Point Message Passing I Communication is between two processors and receiving process must expect a message although can be uncertain asto message type and sending process Information required to specify a message includes Identification of sender process(or) Identification of destination process Type of data -- Integers, Floats, Characaters -- note floating point numbers may need conversion if sending between machines of different type Number of data elements to send and in destination process, maximum size of message that can be received Where the data lives in sending process -- typically a pointer Where the received data should be stored into -- again a pointer Basic Point to Point Message Passing II Two types of communication operations applicable to send and receive Blocking -- sender and receiver wait until communication is complete Non-blocking -- send and receive handled concurrently with computation -- MPI send and receive routines return immediately before message processed so processor can do other things In addition four types of send operation Standard -- A send may be initiated even if matching receive has not been initiated Synchronous -- Waits for recipent to complete and so message delivery guaranteed Buffered -- Copies message into a buffer (if necessary) on destination processor so that completion of send independent of matching receive Ready -- A send is only started when matching receive initiated Blocking Send MPI_Send(C) MPI_SEND(Fortran) call MPI_SEND ( IN message start address of data to send IN message_len number of items (length in bytes determined by type) IN datatype type of each data element IN dest_rank Processor number (rank) of destination IN message_tag tag of message to allow receiver to filter IN communicator Communicator of both sender and receiver group OUT error_message) Error Flag (absent in C) Fortran example: integer count, datatype, dest, tag, comm, mpierr comm= MPI_COMM_WORLD tag=0 datatype=MPI_REAL call MPI_SEND (sndbuf,count,datatype,dest,tag,comm,mpierr) Blocking Receive MPI_Recv(C) MPI_RECV(Fortran) call MPI_RECV( IN start_of_buffer Address of place to store data(address is INput values of data are of course output starting at this adddress!) IN buffer_len Maximum number of items allowed IN datatype Type of each data type IN source_rank Processor number (rank) of source IN tag only accept messages with this tag value IN communicator Communicator of both sender and receiver group OUT return_status Data structure describing what happened! OUT error_message) Error Flag (absent in C) Note that return_status is used after completion of receive to find actual received length (buffer_len is a MAXIMUM), actual source processor rank and actual message tag In C syntax is int error_message=MPI_Recv(void *start_of_buffer,int buffer_len, MPI_DATATYPE datatype, int source_rank, int tag, MPI_Comm communicator, MPI_Status *return_status) Fortran example:Blocking Receive MPI_RECV integer status(MPI_STATUS_SIZE) An array to store status integer mpierr, count, datatype, source, tag, comm integer recvbuf(100) count=100 datatype=MPI_REAL comm=MPI_COMM_WORLD source=MPI_ANY_SOURCE accept any source processor tag=MPI_ANY_TAG accept anmy message tag call MPI_RECV (recvbuf,count,datatype,source,tag,comm,status,mpierr) Note source and tag can be wild-carded Hello World:C Example of Send and Receive #include "mpi.h" main( int argc, char **argv ) { char message[20]; int i, rank, size, tag=137; # Any value of type allowed MPI_Status status; MPI_Init (&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &size); # Number of Processors MPI_Comm_rank(MPI_COMM_WORLD, &rank); # Who is this processor if( rank == 0 ) { # We are on "root" -- Processor 0 strcpy(message,"Hello MPI World"); # Generate message for(i=1; i