The Fortran/PARMACS Array Communication Library

James Allwright University of Southampton

August 3, 1992

1 Introduction

This document describes an implementation of the Array Communication Library in Fortran using PARMACS [?] as the underlying message-passing system. The library was created by porting the original Occam implementation of the library into Fortran and PARMACS. This document does not attempt to explain the use of the library; for this, the reader should consult the documentation for the original Occam libraries [?, ?, ?].

The new library attempts to preserve the structure and format of the old library. However, fundamental differences between Occam and Fortran, and between Channels and the PARMACS messaging system, meant that some changes had to be made :

• Renaming of Routines. Fortran 77 requires identifiers to be no more than 6 characters long and does not allow them to contain the “.” character. Almost all of the original Occam names have been shortened and the new names are given here.

• Variable Length Arrays. Occam allows arrays of unknown size to be passed into a procedure. Their size can then be determined using the SIZE operator. Fortran provides no similar facility, so the length of an array needs to be passed into the Fortran together with the array. In general, the size of an array appears immediately after the array itself in the parameter list. However, for some routines the size is not passed over in this manner as it appears elsewhere in the parameter list.

• Arrays of BYTE. In the original Occam routines, the primitive communication routines pass arrays of bytes. Other types can be passed by RETYPEing them as byte arrays. This is not possible with PARMACS, so the primitive communication routines pass arrays of type INTEGER instead.

• Lack of Dynamic Spawning. PARMACS only allows the host to spawn a set of subordinate processes. These subordinate processes cannot then spawn other processes (as they would be able to in VCR). Hence these libraries have not inherited the dynamic process creation features of the original.

• Communication mechanism. The communication model of Occam is somewhat different to that of PAR-MACS. This is reflected in the way distributed channels have been implemented; in the original Occam they were uni-directional, whereas in this version they are bi-directional.

• Bitwise logical operations. Fortran 77 does not support bitwise logical operations, which Application library I makes use of. This library assumes the existence of intrinsic functions IEOR, IAND and IOR, which will have to be provided if they do not exist in the target Fortran.

There are 4 parts to the ported libraries. Two of these are application level libraries, intended to be used directly by the applications programmer, and two are base level libraries intended mainly to support the application libraries, though they are available to applications programmer.

2 Application Library I : aciapp.u

This library performs synchronization, data re-arrangement (broadcasts and permuations) and partial and global operations (maximum, minimum, sum, product, boolean-or and boolean-and) This library is smaller than the original Occam library as the prefix and global operations are provided only for integer data.

Occam Procedure/Function Fortran Subroutine/Function
SYNCH  SYNCH
BROADCAST  BCAST
BROADCAST.EL  BCSTEL
BROADCAST.ARR  BCSTAR
PERM.DS  PDS
PERM.SHIFT  PSHFT
PERM.REFLECT  PREFL
PERM.XOR  PXOR
PERM.SWAP  PSWAP
PERM.DS.IN  PDSI
PERM.SHIFT.IN  PSHFTI
PERM.REFLECT.IN  PREFLI
PERM.XOR.IN  PXORI
PERM.SWAP.IN  PSWAPI
PREFIX.INT.SUM  INSUM
PREFIX.INT.PROD  INPRD
PREFIX.INT.MAX  INMAX
PREFIX.INT.MIN  INMIN
PREFIX.BOOL.AND  PFXAND
PREFIX.BOOL.OR  PFXOR
GLOB.INT.SUM  GINSUM
GLOB.INT.PROD  GINPRD
GLOB.INT.MAX  GINMAX
GLOB.INT.MIN  GINMIN
GLOB.BOOL.AND  GAND
GLOB.BOOL.OR  GOR

SUBROUTINE SYNCH (rank, rs)
INTEGER rs, rank(0:rs-1)

- synchronises all the processes in the index set supplied in the array rank. This doesn’t terminate on any of the sites connected by rank until it has been called on all of them.

  SUBROUTINE BCAST (x, result, rlen, data, dlen)
  IMPLICIT NONE
  INTEGER x, dlen, data(0:dlen-1), rlen, result(0:dlen-1, 0:rlen-1)

- converts a distribution into a global array. The algorithm is fan-in to origin, followed by broadcast.

SUBROUTINE BCSTEL (x, result, rlen, data, dlen, src)
INTEGER x, rlen, dlen, src, result(0:rlen-1), data(0:dlen-1)
  SUBROUTINE BCSTAR (x, result, rlen, tlen, segsize, data, dlen)
  INTEGER x, rlen, tlen, segsize, dlen
  INTEGER result(0:tlen-1, 0:rlen-1), data(0:tlen-1, 0:dlen-1)

- converts a distribution of arrays into a global array. The algorithm is fan-in to origin, followed by broadcast.

SUBROUTINE PDS (x, result, rlen, data, dlen, src, dst)
INTEGER x, rlen, dlen, result(0:rlen-1), data(0:dlen-1)

- performs a general permutation. The process sends its piece of data to dst and receives a new piece of data from src.

SUBROUTINE PSHFT (x, result, rlen, data, dlen, shift)
INTEGER x, rlen, dlen, shift, result(0:rlen-1), data(0:dlen-1)

- shifts data cyclically in dimension given by index x.

SUBROUTINE PREFL (x, result, rlen, data, dlen, a)
INTEGER x, rlen, dlen, a, result(0:rlen-1), data(0:dlen-1)

- performs a reflection in the dimension given by index x.

SUBROUTINE PXOR (x, result, rlen, data, dlen, p)
INTEGER x, rlen, dlen, p, result(0:rlen-1), data(0:dlen-1)

- performs a generalised butterfly permutation.

      SUBROUTINE PSWAP (x, result, rlen, data, dlen, i1, i2)
      IMPLICIT NONE
      INTEGER x, rlen, dlen, i1, i2, result(0:rlen-1), data(0:dlen-1)

- exchanges data between pairs of processes.

SUBROUTINE PDSI (x, data, dlen, src, dst)
INTEGER x, dlen, src, dst, data(0:dlen-1)

- permutes data array “in situ”; analagous to PDS.

SUBROUTINE PSHFTI (x, data, dlen, shift)
INTEGER x, dlen, shift, data(0:dlen-1)

- permutes data array “in situ”; analagous to PSHFT.

SUBROUTINE PREFLI (x, data, dlen, a)
INTEGER x, dlen, a, data(0:dlen-1)

- permutes data array “in situ”; analagous to PREFL.

SUBROUTINE PXORI (x, data, dlen, p)
INTEGER x, dlen, p, data(0:dlen-1)

- permutes data array “in situ”; analagous to PXOR.

SUBROUTINE PSWAPI (x, data, dlen, i1, i2)
INTEGER x, dlen, i1, i2, data(0:dlen-1)

- permutes data array “in situ”; analagous to PSWAP.

SUBROUTINE INSUM (x, result, data)
INTEGER x, result, data

- calculates a partial sum of integer data in dimension x.

SUBROUTINE INPRD (x, result, data)
INTEGER x, result, data

- calculates a partial product of integer data in dimension x.

SUBROUTINE INMAX (x, result, restag, data, datatg)
INTEGER x, result, restag, data, datatg

- the partial maximum of integer data in dimension x.

SUBROUTINE INMIN (x, result, restag, data, datatg)
INTEGER x, result, restag, data, datatg

- the partial minimum of integer data in dimension x.

SUBROUTINE PFXAND (x, result, data)
INTEGER x
LOGICAL result, data

- performs a partial AND of the boolean data in dimension x.

SUBROUTINE PFXOR (x, result, data)
INTEGER x
LOGICAL result, data

- performs a partial OR of the boolean data in dimension x.

INTEGER FUNCTION GINSUM (x, data)
INTEGER x, data

- calculates a global sum of integer data in dimension x.

INTEGER FUNCTION GINPRD (x, data)
INTEGER x, data

- calculates a global product of integer data in dimension x.

SUBROUTINE GINMAX (res, restg, x, data, datatg)
INTEGER x, data, datatg, res, restg

- calculates a global maximum of integer data in dimension x. It returns res and restg as results.

SUBROUTINE GINMIN (res, restg, x, data, datatg)
INTEGER x, data, datatg, res, restg

- calculates a global minimum of integer data in dimension x. It returns res and restg as results.

LOGICAL FUNCTION GAND (x, data)
INTEGER x
LOGICAL data

- calculates a global AND of boolean data in dimension x.

LOGICAL FUNCTION GOR (x, data)
INTEGER x
LOGICAL data

- calculates a global OR of boolean data in dimension x.

3 Library Accapp - Application comms library

3.1 Introduction

This library supports communication between arrays of processes. The first parameter chan of each routine is the channel which is to be used for communication. The last parameter tag is a single integer value that is passed over with each message. This can be used for signalling some piece of information or for error-checking, much like tagged protocols in Occam. If tag is set to the special value tuntagged = −1, then no tag is sent.

The operation of the input and output routines is not immediately obvious. The routine OUTPUT is called on every process in an array, but causes only a single piece of data to be send over the channel. Thus each process should output the same data and these are merged together. The corresponding INPUT routine broadcasts a single piece of data received on the channel to all processes.

The array input and output routines provide a way of performing the communication so that different processes input or output different things. These routines are supplied with one or more index structures. Processes with the same co-ordinates within those indexes behave identically (e.g. outputting a 1D array to a 2D process array causes the data to be broadcast across one of the dimension). However, processes at different co-ordinates in the indexes can send different data. This data is sent over the channel in an ordered sequence. Likewise, processes at different co-ordinates in the indexes can input different components of an array being sent over a channel.

As there are so many of these routines, no table is provided to show the correspondence between Occam and Fortran routines. However, they follow a straightforward pattern and the naming scheme is systemmatic and relatively simple. Where “INPUT.” appeared in the original name it is replaced by “IN”. Likewise, “OUTPUT.” is replaced by “OU”, “.EL” is replaced by “EL” and “.ARR” is replaced by “AR”. The routines are not all individually described as the behaviour of most of them should be apparent by comparison with other routines.

3.2 Simple I/O

SUBROUTINE INPUT (chan, result, rlen, tag)
INTEGER chan, rlen, tag, result(0:rlen-1)

- inputs global variable result.

SUBROUTINE INTAG (chan, tag)
INTEGER chan, tag

- inputs a tag.

SUBROUTINE OUTPUT (chan, data, dlen, tag)
INTEGER chan, dlen, tag, data(0:dlen-1)

- outputs global variable data.

SUBROUTINE OUTAG (chan, tag)
INTEGER chan, tag

- outputs a tag.

3.3 Process-oriented I/O

SUBROUTINE INDD (chan, rank, rs, result, rlen, tag)
INTEGER chan, rs, rlen, tag, rank(0:rs-1), result(0:rlen-1)

- inputs a whole distribution over the set of indexes contained in the array rank. Each process inputs a single item, which is the array result.

SUBROUTINE IN1D (chan, x, result, rlen, tag)
INTEGER chan, x, rlen, tag, result(0:rlen-1)

- inputs a whole distribution over the index x. Each process inputs a single item, which is the array result.

SUBROUTINE IN2D (chan, x, y, result, rlen, tag)
INTEGER chan, x, y, rlen, tag, result(0:rlen-1)
SUBROUTINE IN3D (chan, x, y, z, result, rlen, tag)
INTEGER chan, x, y, z, rlen, tag, result(0:rlen-1)
SUBROUTINE OUDD (chan, rank, rs, data, dlen, tag)
INTEGER chan, rs, dlen, tag, rank(0:rs-1), data(0:dlen-1)

- outputs a whole distribution over the set of indexes contained in the array rank. Each process outputs a single item, which is the array result.

SUBROUTINE OU1D (chan, x, data, dlen, tag)
INTEGER chan, x, dlen, tag, data(0:dlen-1)
SUBROUTINE OU2D (chan, x, y, data, dlen, tag)
INTEGER chan, x, y, dlen, tag, data(0:dlen-1)
SUBROUTINE OU3D (chan, x, y, z, data, dlen, tag)
INTEGER chan, x, y, z, dlen, tag, data(0:dlen-1)

3.4 Data-oriented I/O

SUBROUTINE INDDAR (chan, rank, rs, ssizes, strides, result, unitlen, tag)
INTEGER chan, rs, unitlen, tag, result(0:*)
INTEGER rank(0:rs-1), ssizes(0:rs-1), strides(0:rs-1)

- inputs an array distributed in the indexes specified in array rank. Each element in the array contains unitlen integers and they are read into the array result. The array strides specifies the change of subscript into result corresponding to incrementing the co-ordinate in each dimension. The array ssizes specifies the number of elements that the calling process will read in each dimension. Thus the total number of items read is the product of the values in sizes. Supplying the extra array strides means that the array result need not be declared to exactly fit the sata it will contain. This is useful when the size of an array is not an exact multiple of the size of the index over which it will be distributed. Also, this routine may be used to input partial arrays by setting up ssizes appropriately.

SUBROUTINE IN1DAR (chan, x, ssx, result, unitlen, ddx, tag)
INTEGER chan, x, ssx, unitlen, ddx, tag, result(0:unitlen-1, 0:ddx-1)

- inputs an array distributed over index x. On the calling process, this routine will input ssx items, each unitlen integers long, into the first ssx elements of result.

SUBROUTINE IN2DAR (chan, y, x, ssy, ssx, result, unitlen, ddy, ddx, tag)
INTEGER chan, x, y, ssx, ssy, unitlen, ddx, ddy, tag
INTEGER result(0:unitlen-1, 0:ddy-1, 0:ddx-1)

- analagous to IN1DAR

SUBROUTINE IN3DAR (chan, x, y, z, ssx, ssy, ssz,
                   result, unitlen, ddx, ddy, ddz, tag)
INTEGER chan, x, y, z, ssx, ssy, ssz, unitlen, tag
INTEGER ddx, ddy, ddz, result(0:unitlen-1, 0:ddx-1, 0:ddy-1, 0:ddz-1)
SUBROUTINE OUDDAR (chan, rank, rs, ssizes, strides, data, unitlen, tag)
INTEGER chan, rs, unitlen, tag, rank(0:rs-1), ssizes(0:rs-1)
INTEGER strides(0:rs-1), data(0:*)
SUBROUTINE OU1DAR (chan, x, ssx, data, unitlen, ddx, tag)
INTEGER chan, x, ssx, unitlen, ddx, tag, data(0:unitlen-1, 0:ddx-1)
SUBROUTINE OU2DAR (chan, y, x, ssy, ssx, data, unitlen, ddy, ddx, tag)
INTEGER chan, x, y, ssx, ssy, unitlen, ddx, ddy, tag
INTEGER data(0:unitlen-1, 0:ddy-1, 0:ddx-1)
SUBROUTINE OU3DAR (chan, x, y, z, ssx, ssy, ssz,
                   data, unitlen, ddx, ddy, ddz, tag)
INTEGER chan, x, y, z, ssx, ssy, ssz, unitlen, tag
INTEGER ddx, ddy, ddz, data(0:unitlen-1, 0:ddx-1, 0:ddy-1, 0:ddz-1)

3.5 I/O for specific data elements

SUBROUTINE INDDEL (chan, rank, rs, result, rlen, dsts, dlen, tag)
INTEGER chan, rs, rlen, dlen, tag
INTEGER rank(0:rs-1), result(0:rlen-1), dsts(0:dlen-1)

- inputs into the array result at the element given by the co-ordinates in array dsts within the indexes in rank. The array dsts should be the same for every process calling it.

SUBROUTINE IN1DEL (chan, x, result, rlen, dstx, tag)
INTEGER chan, x, rlen, dstx, tag, result(0:rlen-1)

- inputs an element at dstx of index x.

SUBROUTINE IN2DEL (chan, x, y, result, rlen, dstx, dsty, tag)
INTEGER chan, x, y, rlen, dstx, dsty, tag, result(0:rlen-1), table(0:1)

- analagous to IN1DEL

SUBROUTINE IN3DEL (chan, x, y, z, result, rlen, dstx, dsty, dstz, tag)
INTEGER chan, x, y, z, rlen, dstx, dsty, dstz, tag
INTEGER result(0:rlen-1)

- inputs element at (dstx, dsty, dstz) of a three-dimensional distribution over x, y, z.

SUBROUTINE OUDDEL (chan, rank, rs, data, dlen, srcs, slen, tag)
INTEGER chan, rs, dlen, slen, tag
INTEGER rank(0:rs-1), data(0:dlen-1), srcs(0:slen-1)

- outputs from the array data on the process at co-ordinates dsts within the indexes in rank. The array dsts should be the same for every process calling it.

SUBROUTINE OU1DEL (chan, x, data, dlen, srcx, tag)
INTEGER chan, x, dlen, srcx, tag, data(0:dlen-1)
SUBROUTINE OU2DEL (chan, x, y, data, dlen, srcx, srcy, tag)
INTEGER chan, x, y, dlen, srcx, srcy, tag, data(0:dlen-1)
SUBROUTINE OU3DEL (chan, x, y, z, data, dlen, srcx, srcy, srcz, tag)
INTEGER chan, x, y, z, dlen, srcx, srcy, srcz, tag, data(0:dlen-1)

4 Base Library I : acibas.u

This is a low-level library of index operations. They are provided mainly for use by the higher level libraries. However, the application programmer is likely to want to use the functions Isize and Icoord directly.

Occam Procedure/Function Fortran Subroutine/Function
INDEX.define  Idefn
INDEX.desc  Idesc
INDEX.size  Isize
INDEX.coord  Icoord
INDEX.in  Iin
INDEX.in (asynchronous) IAin
INDEX.out  Iout
INDEX.out (asynchronous) IAout
INDEX.broadcast  Ibroad
INDEX.merge  Imerge

SUBROUTINE Idefn (index, pids, desc, size, coord)
INTEGER index, desc, size, coord, pids(0:size-1)

Define an index structure.

INTEGER FUNCTION Idesc (index)
INTEGER index

- returns the descriptor of the index. This is a number uniquely specifying the index, which will be between 0 and one less than the number of index structures.

INTEGER FUNCTION Isize (index)
INTEGER index

- returns the size of the index.

INTEGER FUNCTION Icoord (index)
INTEGER index

- returns the co-ordinate of the process within the given index. This will be an integer between 0 and one less than the size of the index.

SUBROUTINE Iin (index, data, dlen, src)
INTEGER index, dlen, data(0:dlen-1), src

- receives the array data from the process at co-ordinate src in the specified index (and at the same co-ordinate as the calling process in all other indexes).

SUBROUTINE IAin (index, data, dlen, src)
INTEGER index, dlen, data(0:dlen-1), src

- an asynchronous version of Iin.

SUBROUTINE Iout (index, data, dlen, dst)
INTEGER index, dlen, data(0:dlen-1), dst

- sends the array data to the process at co-ordinate dst in the specified index (and at the same co-ordinate as the calling process in all other indexes).

SUBROUTINE IAout (index, data, dlen, dst)
INTEGER index, dlen, data(0:dlen-1), dst

- an asynchronous version of Iout.

SUBROUTINE Ibroad (index, data, length, src)
INTEGER index, length, src, data(0:length-1)

- broadcasts the value in data from the process at location src within index to all processes at a different location within that index (but at the same location within other indexes).

SUBROUTINE Imerge (index, data, length, dst, checkglobal)
INTEGER index, dst, length, data(0:length-1)
LOGICAL checkglobal

- causes all processes other than the one at location dst in index send data to dst. This performs a synchronization on that index. If the variable checkglobal is set to .TRUE. then the destination process checks to see that all data received is identical.

5 Base Library II : accbas.u

This library contains low-level routines to handle distributed channels. The routines are primarily for use by the higher-level applications library. Normally, the applications programmer will not use them.

Occam Procedure/Function Fortran Subroutine/Function
IO.REC.set.external  IOwext
IO.REC.set.corank  IOwcor
IO.define  IOdef2
IO.input.define  IOdefn
IO.output.define  IOdefn
IO.rank  IOrank
IO.at.root  IOatR
IO.at.coords  IOatC
IO.local.set  IOlset
IO.root.in  IOrti
IO.forward.in  IOfwdi
IO.root.out  IOrto
IO.forward.out  IOfwdo
IO.in  IOin
IO.tag.in  IOtin
IO.out  IOout
IO.tag.out  IOtout
IO.pre.synch  IOpre
IO.post.synch  IOpost
IO.REC.external  IOrext
IO.REC.corank  IOrcor

SUBROUTINE IOwext( data, extpid)
INTEGER data(0:IOmemsize-1), extpid

- writes the value extpid into the channel’s external processor id field.

SUBROUTINE IOwcor(data, corank, dim)
INTEGER data(0:IOmemsize-1), corank(0:dim-1)

- writes the array corank in the corank field of a channel.

SUBROUTINE IOdefn (chan, type, extpid, corank, length)
INTEGER chan, type, extpid, length, corank(0:length-1)

- procedure creates a channel structure.

SUBROUTINE IOrank (chan, rank, rs)
INTEGER chan, rs, rank(0:rs-1)

- defines the rank to be used prior to a data transfer taking place.

SUBROUTINE IOatR (chan, atroot)
INTEGER chan
LOGICAL atroot

- sets the variable atroot to be true or false depending on whether the calling process is at the root of the specified channel.

SUBROUTINE IOatC (chan, atcoords, coords, clen)
INTEGER chan, clen, coords(0:clen-1), i, ranksize
LOGICAL atcoords

- sets the variable atccords to true or false depending on whether the calling process is at the specified coordinates.

SUBROUTINE IOlset (chan, locset, glset, length)
INTEGER chan, length, locset(0:1, 0:length-1), glset(0:1, 0:length-1)

This routine is provided for optimisation of procedures for inputting or outputting multiple elements of a distribution. glset defines a volume of sites from which elements are to be the input or output. Both it and locset are arrays sized according to the current rank. Each element defines a coordinate range by a pair – a base and count. The locset returned is a subset of glset, namely that set of sites for which the local process may have to perform forwarding. If the current transfer is to or from a site which is in the global set but not the local set, the local process needn’t engage in a call to the corresponding “forward” operation.

SUBROUTINE IOrti  (chan, result, length)
INTEGER chan, length, result(0:length-1)

- performs input on the root process.

SUBROUTINE IOfwdi (chan, data, dlen, coords, clen)
INTEGER chan, dlen, clen, data(0:dlen-1), coords(0:clen-1)

- broadcasts the value of data from root along the forwarding path to the sites with coords relative to the current rank. The size of the data array must be global across all the processes involved.

SUBROUTINE IOrto  (chan, data, length)
INTEGER chan, length, data(0:length-1)

- performs an output on the root process.

SUBROUTINE IOfwdo (chan, data, dlen, coords, clen)
INTEGER chan, dlen, clen, data(0:dlen-1), coords(0:clen-1)

- forwards data prior to it being output by the root process.

SUBROUTINE IOin (chan, result, rlen, coords, clen)
IMPLICIT NONE
INTEGER chan, rlen, clen, result(0:rlen-1), coords(0:clen-1)

- performs input on a channel.

SUBROUTINE IOtin (chan, tag)
INTEGER chan, tag

- inputs a tag on a channel.

SUBROUTINE IOout (chan, data, dlen, coords, clen)
INTEGER chan, dlen, clen, data(0:dlen-1), coords(0:clen-1)

- outputs on a channel.

SUBROUTINE IOtout (chan, tag)
INTEGER chan, tag

- outputs a tag on a channel.

SUBROUTINE IOpre (chan)
INTEGER chan

- performs partial synchronization. This guarantees not to terminate on the root until it has been called on all processes.

SUBROUTINE IOpost (chan)
INTEGER chan

- performs partial synchronization. This guarantees not to terminate on any site until after it has been called at the root.

SUBROUTINE IOrext (data, external)
INTEGER data(0:IOmemsize-1), external

- returns the value in the external id field of a channel.

SUBROUTINE IOrcor (data, coranksize, corank, length)
INTEGER data(0:IOmemsize-1), coranksize, length, corank(0:length-1)

- returns the corank field of a channel.

6 Configuration routines

The array communication library supports an array-based parallel computation model. The user’s program consists of multidimensional arrays of identical processes; a typical configuration might be a host (conceptually a zero-dimensional array) and a 2-D grid of worker processes. Associated with each dimension is an object called an index. This has only one important attribute, a size, which is the number of processes in that dimension. In addition, there are objects called channels which allow communication between arrays of processes. Unlike Occam channels and unlike the channels in the Occam version of this library, these channels allow bi-directional communication.

The user’s program consists of code for the processes that are to be run and a configuration level that initiates these processes. Below is an example configuration level :

INTEGER xid, yid, zid
INTEGER xsize, ysize, zsize
INTEGER hostarr, gridarr, linearr
INTEGER griddim(3), linedim(1)
INTEGER hostch(1), gridch(2), linech(1)
PARAMETER (xid   = 0, yid   = 1, zid   = 2)
PARAMETER (xsize = 4, ysize = 4, zsize = 4)
PARAMETER (hostarr = 0, gridarr = 1, linearr = 2)
PARAMETER (hgchan = 0, glchan = 1)
DATA griddim /xid, yid, zid/, linedim /xid/

DATA hostch /hgchan/, gridch /hgchan, glchan/, linech /glchan/
CALL acbegin ()
CALL acindx (xid, xsize)
CALL acindx (yid, ysize)
CALL acindx (zid, zsize)
CALL acchan (hgchan, 0)
CALL acchan (glchan, 1, linedim)
CALL acarry (hostarr, 1, hostch, 0)
CALL acarry (gridarr, 2, gridch, 3, griddim)
CALL acarry (linearr, 2, linech, 3, linedim)

The routines are :

SUBROUTINE acbegin()

- performs housekeeping tasks. This should be the first configuration routine to be called.

SUBROUTINE acindx(label, size)
INTEGER label, size

- sets up an index structure of size size. The label is a number uniquely indentifying that index, used solely within the configuration level.

SUBROUTINE acchan(name, dimension, indexes)
INTEGER dimension, indexes(dimension)

- sets up a channel or array of channels. If only one channel is to be created, the parameter dimension is set to zero and the last parameter may be ommited. If an array of channels is required, indexes should be an array of dimension channels. The number of parallel created is the product of the sizes of the indexes in this array.

SUBROUTINE accarry(subroutine, nchans, channels, nindexes, indexes)
INTEGER nchans, nindexes, chans(nchans), indexes(nindexes)

- spawns a process array. The number of processes spawned is the product of the sizes of the indexes. The parameter subroutine is the name of the routine to be spawned. This subroutine must have a parameter list of

the form nchans, channels, nindexes, indexes, where channels is an array of nchans channels and indexes

is an array of nindexes indexes.

SUBROUTINE acend()

- performs housekeeping tasks. This should be the last configuration routine to be called.

7 Example: matrix transpose

Below is an example program using the array communication library. The configuration level sets up a master process and a 2-D array of 3 by 3 processes. The master sets up a 3 by 3 matrix and sends it to the workers. The workers transpose it using three “in situ” permuation operations and send it back to the master, which prints it out.

C master
      SUBROUTINE display (source, rows, columns)
C  Read a stream of 'rows' * 'columns' real numbers from 'source',
C  and print as a table.
      IMPLICIT NONE
      INTEGER source, rows, columns
      INTEGER i, j, tag, data(0:40)
      INTEGER tend, tint, tr32, tstr, tuntagged
      PARAMETER (tuntagged = -1)

PARAMETER (tend = 0)
PARAMETER (tint = 1)
PARAMETER (tr32 = 2)
PARAMETER (tstr = 3)
DO 10, i = 0, rows-1
  DO 20, j = 0, columns-1
     call INTAG(source, tag)
     IF (tag .EQ. tr32) THEN
       call INPUT(source, data(j), 1, tuntagged)
     ELSE
       STOP "Unexpected Tag value"
     ENDIF
20      CONTINUE
  print '(50(i3, ''  ''))', (data(j), j = 0, columns-1)
10    CONTINUE
END
C
SUBROUTINE transpose (chan, size)
IMPLICIT NONE
INTEGER chan, size, i, j
INTEGER tend, tint, tr32, tstr, tuntagged
PARAMETER (tuntagged = -1)
PARAMETER (tend = 0)
PARAMETER (tint = 1)
PARAMETER (tr32 = 2)
PARAMETER (tstr = 3)
DO 30, i = 0, size-1
  DO 40, j = 0, size-1
    call OUTAG(chan, tr32)
    call OUTPUT(chan, ((size * i) + j), 1, tuntagged)
40      CONTINUE
30    CONTINUE
print *, 'matrix'
call display(chan, size, size)
print *, 'transpose'
call display (chan, size, size)
END
C
SUBROUTINE master (nchans, chan)
IMPLICIT NONE
INTEGER nchans
INTEGER chan
INTEGER tend, tint, tr32, tstr, tuntagged
PARAMETER (tuntagged = -1)
PARAMETER (tend = 0)
PARAMETER (tint = 1)
PARAMETER (tr32 = 2)
PARAMETER (tstr = 3)
INTEGER size, tag
50    CONTINUE
call INTAG(chan, tag)
IF (tag .EQ. tint) THEN
  call INPUT(chan, size, 1, tuntagged)
  print *, 'Calling tranpose'
  call transpose(chan, size)
  GOTO 50
ENDIF
END

C worker
SUBROUTINE worker (nchans, chan, dim, rank)
IMPLICIT NONE
INTEGER dim, nchans
INTEGER chan
INTEGER rank (0 : dim - 1)
INTEGER x, y
INTEGER Isize, Icoord
INTEGER tend, tint, tr32, tstr, tuntagged
PARAMETER (tuntagged = -1)
PARAMETER (tend = 0)
PARAMETER (tint = 1)
PARAMETER (tr32 = 2)
PARAMETER (tstr = 3)
INTEGER L, A
x = rank(0)
y = rank(1)
C  3 * 3 problem (one element per process).
L = Isize (x)
call OUTPUT (chan, L, 1, tint)
call IN2D (chan, x, y, A, 1, tr32)
call OU2D (chan, x, y, A, 1, tr32)
call PSHFTI (x, A, 1, Icoord (y))
call PREFLI (y, A, 1, Icoord (x))
call PSHFTI (x, A, 1,-Icoord (y))
call OU2D (chan, x, y, A, 1, tr32)
call OUTAG (chan, tend)
END
C
C configuration
IMPLICIT NONE
EXTERNAL master, worker
INTEGER xid, yid
INTEGER xsize, ysize
INTEGER chan, null
INTEGER rank (2)
PARAMETER (xid   = 0, yid   = 1)
PARAMETER (xsize = 3, ysize = 3)
PARAMETER (chan = 0)
DATA rank /xid, yid/
CALL acbegn ()
CALL acindx (xid, xsize)
CALL acindx (yid, ysize)
CALL acchan (chan, 0)
CALL acarry (master, 1, chan, 0, null)
CALL acarry (worker, 1, chan, 2, rank)
CALL acdone ()
END