
\documentclass[runningheads]{llncs}

\usepackage{psfig}

\begin{document}

\pagestyle{headings}

\bibliographystyle{plain}

\mainmatter

\title{ARMCI: A Portable Remote Memory Copy Library for Distributed Array
Libraries and Compiler Run-time Systems}

\titlerunning{Lecture Notes in Computer Science}

\author{Jarek Nieplocha\inst{1} and Bryan Carpenter\inst{2}}

\authorrunning{Jarek Nieplocha et al.}

\institute{Pacific Northwest National Laboratory, \\ Richland, WA 99352 \\
           \email{j\_nieplocha@pnl.gov}
\and
           Northeast Parallel Architectures Center, \\ Syracuse University \\
           \email{dbc@npac.syr.edu}
}

\date{January 8, 1998}

\maketitle

\begin{abstract}
This paper introduces a new portable communication library called
ARMCI.  ARMCI provides one-sided communication capabilities for
distributed array libraries and compiler run-time systems.  It supports
remote memory copy, accumulate, and synchronization operations
optimized for non-contiguous data transfers including strided and
generalized UNIX I/O vector interfaces.  The library has been employed
in the Global Arrays shared memory programming toolkit and Adlib,
a Parallel Compiler Run-time Consortium run-time system.
\end{abstract}

\section{Introduction}
\label{sec:introduction}

A portable lightweight remote memory copy is useful in parallel
distributed-array libraries and compiler run-time systems.  This
functionality allows any process of a MIMD program executing in
a distributed memory environment to copy data between local
and remote memory.  In contrast to the message-passing model used
predominantly in these environments, remote memory copy does not require
the explicit cooperation of the remote process whose memory is accessed.
By decoupling synchronization between the process that needs the data and
the process that owns the data from the actual data transfer, implementation
of parallel algorithms that operate on distributed and irregular data
structures can be greatly simplified and performance improved.  In
distributed array libraries, remote memory copy can be used for
several purposes, incuding implementation of the object-based shared memory
programming environment of Global Arrays \cite{GA} or ABC++ \cite{Eigler96},
and optimized implementation of collective operations on sections of
multidimensional distributed arrrays, like those in the PCRC Adlib
run-time system \cite{NPAC_PCRC_kernel} or the P++ \cite{Quinlan94}
object oriented class library.

In scientific computing, distributed array libraries communicate
sections or fragments of sparse or dense multidimensional arrays.  With
remote copy APIs (application programming interfaces) supporting only
contiguous blocks of data, array sections are typically decomposed into
contiguous blocks of data and transferred in multiple communication
operations.  This approach is quite inefficient on systems with high
latency networks, as the communication subsystem would handle each
contiguous portion of the data as a separate network message.  The
communication startup costs are incurred multiple times rather than just
once.  The problem is attributable to an inadequate API, which does not
pass enough information about the intended data transfer and layout of
user data to the communication subsystem.  In principle, if a more
descriptive communication interface is used, there are are many ways
communication libraries could optimize performance.  For example:
1) minimize the number of underlying network packets by packing
distinct blocks of data into as few packets as possible, 2) minimize
the number of interrupts in interrupt-driven message-delivery systems,
and 3) take advantage of any available shared memory optimizations
(prefetching, poststoring, bulk data transfer) on shared memory systems.
We illustrate the performance implication of communication interfaces
with an example.

Consider a {\em get} operation that transfers a $2 \times 100$ section
of a $10 \times 300$ Fortran array of double precision numbers on the
IBM SP.  The native remote memory copy operation {\em LAPI\_Get}
supports only contiguous data transfers \cite{Shah98,LAPI}.  Therefore,
100 calls are required to transfer data in 16-byte chunks.  With
90$\mu$s latency for the first column and 30$\mu$s for the next ones
(pipelining), the transfer rate is 0.5MB/s.  For the same amount of
contiguous data, the rate is 30MB/s.  If the application used
a communication interface supporting strided data layout, a communication
library could send one LAPI active message specifying the entire
request.  At the remote end the AM handler could copy data into a
contiguous buffer and send all data in a single network packet.
The data would then be copied to the destination.  The achieved rate
is 9MB/s.  We argue that platform-specific optimizations such as these
should be made available through a portable communication library, rather
than being reproduced in each distributed array library or run-time system
that requires strided memory copy.

In this paper, we describe design, implementation and experience with the
ARMCI (Aggregate Remote Memory Copy Interface), a new portable remote
memory copy library we have been developing for optimized communication
in the context of distributed arrays.  ARMCI aims to be fully portable
and compatible with message-passing libraries such as MPI or PVM.
Unlike most existing similar facilities, such as Cray SHMEM \cite{Barriuso94}
or IBM LAPI \cite{LAPI}, it focuses on the noncontiguous data
transfers.  ARMCI offers a simpler and lower-level model of one-sided
communication than MPI-2 \cite{MPI2} (no epochs, windows, datatypes,
Fortran-77 API, rather complicated progress rules, etc.) and targets
a different user audience.  In particular, ARMCI is meant to be used
primarily by library implementors rather than application developers.
Examples of libraries that ARMCI is aimed at include Global Arrays,
P++/Overture \cite{Quinlan94,Bassetti98}, and the PCRC Adlib run-time system.
We will discuss applications of ARMCI in Global Arrays and Adlib as
particular examples.  Performance benefits from using ARMCI are presented
by comparing its performance to that of the native remote memory copy
operation on the IBM SP.

\section{ARMCI Model and Implementation}
\label{sec:armci}

\subsection{ARMCI Operations}

ARMCI supports three classes of operation
\begin{itemize}
\item
data transfer operations including put, get and accumulate.
\item
synchronization operations---local and global fence and atomic
read-modify-write, and
\item
utility operations for allocation and deallocation of memory
(as a convenience to the user) and error handling.
\end{itemize}
The currently supported operations are listed in Table \ref{tab:ops}.
\begin{table}[tb]
\caption{ARMCI operations\label{tab:ops}}
\small
\centerline{
\begin{tabular}{| c | c |}
\hline
{\normalsize Operation} & {\normalsize Comment} \\
\hline \hline
{\em ARMCI\_Put}, {\em ARMCI\_PutV}, &
contiguous, vector and strided versions of put \\
{\em ARMCI\_PutS} & \\
\hline
{\em ARMCI\_Get}, {\em ARMCI\_GetV}, &
contiguous, vector and strided versions of get \\
{\em ARMCI\_GetS} & \\
\hline
{\em ARMCI\_Acc}, {\em ARMCI\_AccV},  &
contiguous, vector and strided versions of \\ 
{\em ARMCI\_AccS} & atomic accumulate \\
\hline
{\em ARMCI\_Fence} &
blocks until outstanding operations targeting \\ &
specified process complete \\
\hline
{\em ARMCI\_AllFence} &
blocks until all outstanding operations issued \\ &
by calling process complete \\
\hline
{\em ARMCI\_Rmw} &
atomic read-modify-write \\
\hline
{\em ARMCI\_Malloc} &
memory allocator, returns array of addresses \\ &
for memory allocated by all processes \\
\hline
{\em ARMCI\_Free} &
free memory allocated by {\em ARMCI\_Malloc} \\
\hline
{\em ARMCI\_Poll} &
polling operation, not required for progress \\
\hline
\end{tabular}
}
\end{table}

The data transfer operations are available with two noncontiguous
data format:
\begin{enumerate}
\item
{\em Generalized I/O vector.}  This is the most general format intended
for multiple sets of equally-sized data segments, moved between arbitrary
local and remote memory locations.  It extends the format used in the
UNIX {\em readv/writev} operations by minimizing storage requirements
in cases when multiple data segments have the same size.  For example,
operations that would map well to this format include scatter and gather.
The format would also allow to transfer a triangular section of a 2-D
array in one operation.
\small
\begin{verbatim}
    typedef struct {
      void *src_ptr_ar ;
      void *dst_ptr_ar ;
      int bytes ;
      int ptr_ar_len ;
    } armci_giov_t ;
\end{verbatim}
\normalsize
For example, with the generalized I/O vector format a {\em put} operation
that copies data to the process(or) {\em proc}'s memory has the following
interface:
\small
\begin{verbatim}
    int ARMCI_PutV(armci_giov_t dscr_arr[], int len, int proc)
\end{verbatim}
\normalsize
The first argument is an array of size {\em arr\_len}.  Each array element
specifies a set of equally-size segments of data copied from the local
memory to the memory at the remote process(or) {\em proc}.
\begin{figure}[tb]
%\centerline{\psfig{figure=figure1.eps,width=4in}}
\centerline{\psfig{figure=figure1.eps}}
\caption{Descriptor array for generalized I/O vector format.  Each
array element describes transfer of a set of equally-sized ({\em bytes})
blocks of data.\label{fig:generalized}}
\end{figure}

\item
{\em Strided.}  This format is an optimization of the generalized I/O
vector format for sections of dense multidimensional arrays.  Instead
of including addresses for all the segments, it specifies for source
and destination only the address of the first segment in the set.  The
addresses of the other segments are computing using the stride
information:
\small
\begin{verbatim}
    void *src_ptr ;
    void *dst_ptr ;
    int stride_levels ;
    int src_stride_arr[stride_levels] ;
    int dst_stride_arr[stride_levels] ;
    int count[stride_levels + 1] ;
\end{verbatim}
\normalsize
For example, with the strided format a {\em put} operation that copies
data to the process(or) {\em proc}'s memory has the following interface:
\small
\begin{verbatim}
    int ARMCI_PutS(src_ptr, src_stride_ar, dst_ptr,
                   dst_stride_ar, count, stride_levels, proc)
\end{verbatim}
\normalsize
The first argument is an array of size {\em arr\_len}.  Each array element
specifies a set of equally-sized segments of data to be copied from the
local memory to the memory at the remote process(or) {\em proc}, see Figure
\ref{fig:strided}.

\begin{figure}[tb]
%\centerline{\psfig{figure=figure2.eps,width=4in}}
\centerline{\psfig{figure=figure2.eps}}
\caption{Strided format for 2-dimensional array sections (assuming
Fortran layout).\label{fig:strided}}
\end{figure}

\end{enumerate}

For contiguous data, {\em put} and {\em get} operations are available
as simple macros on top of their strided counterparts.

\subsection{Atomic Operations}

At the present time, ARMCI offers two atomic operations: {\em accumulate}
and {\em read-modify-write}.

{\em Accumulate} is an important operation in many scientific codes.
It combines local and remote data atomically $x = x + a \times y$.
For double precision data types, it can be thought of as an atomic
version of the BLAS {\em DAXPY} subroutine with the $x$ array located
in remote memory.  Unlike {\em MPI\_Accumulate} in MPI-2, the ARMCI
{\em accumulate} preserves the scale argument $a$ available in {\em DAXPY}.
For applications that need scaling, it has performance advantages on many
modern microprocessors, where thanks to the {\em multiply-and-add} operation
for floating point datatypes, the scaled addition is executed in the same
time as the addition operation alone.  The ARMCI {\em accumulate} is
available for integer, double precision, complex and double complex
datatypes.

{\em ARMCI\_Rmw} is another atomic operation in the library.  It can be used
to implement synchronization operations or as a shared memory counter
in simple dynamic load balancing applications.  The operation updates
atomically a remote integral variable according to the specified operator
and returns the old value.  There are two operators available:
{\em fetch-and-increment} and {\em swap}.

\subsection{Progress and ordering}

It is important for a communication library such as ARMCI to have
straightforward and uniform progress rules on all supported platforms.
They simplify development and performance analysis of applications and
free the user from dealing with ambiguities of platform-specific
implementations.  On all platforms, the ARMCI operations are truly
one-sided and complete regardless of the actions taken by the remote
process(or).  In particular, there is no need for the remote process to
make occasional communication calls or poll in order to assure that
communication calls issued by other processes(ors) to this process(or)
can complete.  Although, for performance reasons, polling can be
helpful (can avoid the cost of interrupt processing), it is not
necessary to assure progress \cite{Langendoen96}.

The ARMCI operations are ordered (complete in order they were issued)
when referencing the same remote process(or).  Operations issued to
different processors can complete in an arbitrary order.  Ordering
simplifies the programming model and is required in many applications
in computational chemistry (for example).  Some systems allow ordering
of otherwise unordered operations by providing a fence operation that
blocks the calling process until the outstanding operation completes,
so the next operation issued does not overtake it.  This approach
usually accomplishes more than applications might desire, and could
have a negative performance impact on platforms where the copy
operations are otherwise ordered.  Usually, ordering can be
accomplished with a lower overhead by using platform-specific means
inside the communication library, rather than by a fence operation at
the application level.

In ARMCI, when a {\em put} or {\em accumulate} operation completes,
the data has been copied out of the calling process(or) memory but
has not necessarily arrived at its destination.  This is a local completion.
A global completion of the outstanding put operations can be achieved
by calling {\em ARMCI\_Fence} or {\em ARMCI\_AllFence}. {\em ARMCI\_Fence}
blocks the calling processor until all {\em put} operations issued to a
specified remote process(or) complete at the destination. 
{\em ARMCI\_AllFence} does the same for all outstanding {\em put} operations
issued by the calling process(or), regardless of the destination.

\subsection{Portability and implementation considerations}

The ARMCI model specification does not describe or assume any particular
implementation model (for example, threads).  One of the primary design
goals was to allow wide portability of the library.  Another one was to
allow an implementation to exploit the most efficient mechanisms available
on a given platform, which might involve active messages [11], native
put/get, shared memory, and/or threads.

For example, depending on whatever solution delivers the best performance,
ARMCI {\em accumulate} might or might not be implemented using the
``owner computes'' rule.  In particular, this rule is used on IBM SP
where {\em accumulate} is executed inside the Active Message handler on
the processor that owns the remote data whereas on the Cray T3E the
requesting processor performs the calculation (by locking memory,
copying data, accumulating, putting updated data back and unlocking)
without interrupting the data owner.

On shared memory systems, ARMCI operations currently require the remote
data to be located in shared memory.  This restriction greatly simplifies
the implementation and improves the performance by allowing the library
to use simply an optimized memory copy rather than other more costly
mechanisms for moving data between separate address spaces.  This is
acceptable in the distributed array libraries we considered since they
are responsible for the memory allocation and can easily use
{\em ARMCI\_Malloc} (or other mechanisms such as using {\em mmap},
{\em shmget}, etc. directly) rather than the local memory allocation.
However, the requirement can be lifted in future version of ARMCI if
compelling reasons are identified.

At the present time, the library is implemented on top of existing
native communication interfaces on:
\begin{itemize}
\item
distributed-memory systems such as Cray T3E and IBM SP,
\item
shared memory systems such as SGI Origin and Cray J90, and Unix and
Windows NT workstations.
\end{itemize}

On distributed memory platforms, ARMCI uses the native remote memory
copy (SHMEM on Cray T3E and LAPI on IBM SP).  In addition, on IBM SP
LAPI active messages and threads are employed to optimize performance
of noncontiguous data transfers for all but very small and very large
messages.  As an example, in Figure \ref{fig:SPImplementation} we
demonstrate the implementation of the {\em strided get} on the IBM
SP.  Implementations of {\em strided put} and {\em accumulate}
are somewhat more complex due to the additional level of latency
optimization for short active messages and mutual exclusion in
{\em accumulate}.  The implementation of vector operations is close
to that of their strided counterparts on the IBM SP.  Mutual
exclusion necessary to implement atomic operations is implemented
with the Pthread mutexes on the IBM SP and the atomic swap on the Cray
T3E.

\begin{figure}[tb]
%\centerline{\psfig{figure=figure3.eps,width=4in}}
\centerline{\psfig{figure=figure3.eps}}
\caption{Implementation of the ARMCI strided get operation on the IBM SP
\label{fig:SPImplementation}}
\end{figure}

The ARMCI ports on shared memory systems are based on the shared memory
operations (Unicos shared memory on the Cray J90, System V on other
Unix systems, and memory mapped files on Windows NT) and optimized remote
memory copies for noncontiguous data transfers.  On most of the platforms,
the best performance of the memory copy is achieved with the Fortran-77
compiler.  Many implementations of the {\em memcpy} or {\em bcopy}
operations in the standard C library are far from optimal even for
well aligned contiguous data.  Mutual exclusion is implemented using
vendor-specific locks (on SGI, HP, Cray-J90), System V semaphores
(Unix) or thread mutexes (NT).

The near-future porting plans for ARMCI include clusters of
workstations, and Fujitsu distributed memory supercomputers on top
of the Fujitsu MPlib low-level communication interface.

\subsection{Performance}

We demonstrate performance benefits of ARMCI by comparing it to performance
of the native remote memory copy on the IBM SP in two contexts: 1) copying
sections of 2-D arrays from remote to local memory and 2) gather operation
for multiple double-precision elements.  In both cases, we run the tests
on four processors with processor 0 alternating its multiple requests
between the other three processors which were sleeping at that time.
This assured that the incoming requests were processed in the interrupt
mode (worst case scenario).  We performed these tests on a 512-node IBM
SP at PNNL.  The machine uses the 120MHz Power2 Super processor and
512MB main memory on each uniprocessor node, and the IBM high performance
switch (with the TB-3 adapter).

In the first benchmark, we transferred square sections of 2-dimensional
arrays of double precision numbers from remote to local memory (get operation)
and varied the array section dimension from 1 to 512.  The array section
was transferred using either a strided {\em ARMCI\_GetS} operation or
multiple {\em LAPI\_Get} operations that transferred contiguous blocks
of data---columns in the array section.  The latter implementation
uses nonblocking {\em LAPI\_Get} operations which allows to issue requests
for all columns in the section and then wait for all of them to complete.
Had we used the blocking operation waiting for data in each column to
arrive before issuing another {\em get} call the performance would have
been much worse.  Even with this optimization, Figure \ref{fig:GetSPerf}
shows that the ARMCI strided {\em get} operation has very significant
performance advantages over the native contiguous {\em get} operation
when transferring array sections.  There are two exceptions where the
performances of both approaches are identical:
\begin{itemize}
\item
for small array sections where {\em ARMCI\_GetS} uses {\em LAPI\_Get}
before switching to the Active Message (AM) protocol to optimize
latency, and
\item
very large array sections where {\em ARMCI\_GetS} switches again from
the Active Message to {\em LAPI\_Get} protocol to optimize bandwidth
when the cost of the two extra memory copies makes the AM protocol
less competitive, see Figure \ref{fig:SPImplementation}.
\end{itemize}

\begin{figure}[tb]
%\centerline{\psfig{figure=figure4.eps,width=4in}}
\centerline{\psfig{figure=figure4.eps}}
\caption{The log-linear (left) and linear-linear (right) graphs represent
bendwidths for a contiguous get operation and ARMCI {\em strided} get
when moving square sections of a 2-D array.
\label{fig:GetSPerf}}
\end{figure}

In Figure \ref{fig:GetVPerf}, performances of the {\em gather} operation
for double precision elements is presented.  In particular the {\em gather}
operation in this test fetches every third element of an array on a
remote processor.  For comparision, the operation is implemented on top
of the ARMCI vector {\em get} operation, {\em ARMCI\_GetV}, and multiple
native {\em get} operations.  In this case too we used nonblocking
{\em LAPI\_Get} followed by a single wait operation to improve performance
of this implementation.  Despite this optimization, the performance of
ARMCI is by far better for all but very small requests, for which the
performances of both approaches are identical.  For small requests
{\em ARMCI\_GetV} is implemented using multiple nonblocking {\em LAPI\_Get}
operaitons up to the point where the Active Message based protocol
becomes more competitive.  This protocol is very similar to that for
the strided get shown in Figure \ref{fig:SPImplementation}.  The
performance of {\em gather} implemented on top of the nonblocking native
{\em get} operation can be explained by the pipelining effect \cite{Shah98}
for multiple requests in LAPI.  Only the first {\em get} request issued
to a particular remote processor is processed in the interrupt mode,
the following ones are received when processing has not been completed.
Regardless of pipelining, on eight bytes of user data is sent in each
{\em LAPI\_Get} message (single network packet) whereas in the case of
{\em ARMCI\_GetV} all network packets sent by {\em LAPI\_Put} inside
the AM handler are filled with user data.  With the 1024-byte packets
in the IBM SP network, the network utilization and performance of the
{\em gather} operation on top of the native {\em get} operation are far
from optimal.

\begin{figure}[tb]
%\centerline{\psfig{figure=figure5.eps,width=4in}}
\centerline{\psfig{figure=figure5.eps}}
\caption{The log-log (left) and linear-log (right) graphs represent
bendwidth in gather operations for double precision numbers as a
function of the number of elements when implemented with contiguous
get operations and the ARMCI {\em vector} get.
\label{fig:GetVPerf}}
\end{figure}

\section{Experience with ARMCI}

\subsection{Global Arrays}

The Global Arrays (GA) \cite{GA} toolkit provides a shared-memory
programming model in the context of 2-dimensional distributed arrays.
GA has been the primary programming model for numerous applications, some
as big as 500,000 lines of code, in quantum chemistry, molecular dynamics,
financial calculations and other areas.  As part of the DoE-2000 Advanced
Computational Testing and Simulation (ACTS) toolkit, GA is being
extended to support higher dimensional-arrays.  The toolkit was originally
implemented directly on top of system-specific communication mechanisms
(NX {\em hrecv}, MPL {\em rcvncall}, SHMEM, SGI {\em arena}, etc.).
The original one-sided communication engine of GA had been closely
tailored to the two-dimensional array semantics supported by the toolkit.
This specificity hampered extensibility of the GA implementation.  It
became clear that a separation of the communication layer from the
distributed array infrastructure is a much better approach.  This was
accomplished by restructuring the GA toolkit to use ARMCI hence making
the GA implementation itself fully platform independent.  ARMCI matches
the GA model well and allows GA to support arbitrary dimensional
arrays efficiently.  The overhead introduced by using an additional
software layer (ARMCI) is small; for example on the Cray T3E {\em ga\_get}
latency has increased by less than 0.5$\mu$S when comparing with the
original implementation of GA.


\subsection{Adlib}

The Adlib library was completed in the Parallel Compiler Runtime
Consortium project \cite{Common_runtime} project.  It is a high-level
runtime library designed to support translation of data-parallel
languages \cite{NPAC_PCRC_kernel}.  Initial emphasis was on High
Performance Fortran, and two experimental HPF translators used the
library to manage their communications \cite{Zhang97,merlin96}.
Currently the library is being used in the {\em HPspmd} project at NPAC
\cite{Carpenter98}.  It incorporates a built-in
representation of a distributed array, and a library of communication
and arithmetic operations acting on these arrays.  The array model is
more general than GA, supporting general HPF-like distribution formats,
and arbitrary regular sections.  The existing Adlib communication
library emphasizes collective communication rather than one-sided
communication.

The kernel Adlib library is implemented directly on top of MPI.
We are reimplementing parts of the collective communication library on
top of ARMCI and expect to see improved performance on shared memory
platforms.  The layout information must now be accompanied by
a locally-held table of pointers containing base addresses at
which array segments are stored in each peer process (in message-passing
implementations, only the local segment address is needed).  Adlib
is implemented in C++, and for now the extra fields are added by
using inheritance with the kernel distributed array descriptor class
(DAD) as base class.  A representative selection of the Adlib communication
schedule classes were reimplemented in terms of ARMCI.  These included
the {\em Remap} class, which implements copying of regular sections
between distributed arrays, the {\em Gather} class, which implements
copying of a whole array from a source array indirectly subscripted
by some other distributed arrays, and a few related classes.

The benchmark presented here is based on the {\em remap} operation.
The particular example chosen abstracts the communication in the
array assignment of the following HPF fragment
\small
\begin{verbatim}
          real a(n, n), b(n, n)
    !hpf$ distribute a(block, *) onto p
    !hpf$ distribute b(*, block) onto p
          a = b
\end{verbatim}
\normalsize
The destination array is distributed in its first dimension and the
source array is distributed in its second dimension, so the assignment
involves a data redistribution requiring an all-to-all communication.
In practice the benchmark was coded directly in C++ (rather than Fortran)
and run on four processors of an SGI Power Challenge.  The timings for
old and new versions are given in Figure \ref{fig:AdlibPerf}.
The initial implementation of the operation on top of ARMCI already
shows a modest but significant 20\% improvement over MPI for large
arrays.  For small arrays the MPI version was actually slightly
faster due to extra barrier synchronizations used in the initial
ARMCI implementation of {\em remap} (for now these synchronizations
were implemented naively in terms of message-passing).  The MPI implemenation
is MPICH using the shared memory device, so the underlying transport is
the same in both cases.

\begin{figure}[tb]
%\centerline{\psfig{figure=figure6.eps,width=4in}}
\centerline{\psfig{figure=figure6.eps}}
\caption{Timings for original MPI vs new ARMCI implementation of {\em remap}.
The operation is a particular redistribution of an $N$ by $N$ array.
\label{fig:AdlibPerf}}
\end{figure}

In addition to improved performance of collective operations on shared
memory platforms, adopting ARMCI as a component of the Adlib implementation
will enable the next release of the library to incorporate one-sided
{\em get} and {\em put} operations as part of its API---a notable
gap in the existing functionality.  In a slightly orthogonal
but complementary development, we have produced a PCRC version of GA
that internally uses a PCRC/Adlib array descriptor to maintain the global array
distribution parameters.  This provides a GA style programming model,
but uses the Adlib array descriptor internally.  The modularity of
the GA implementation, enhanced by the introduction of the ARMCI
layer, made changing the internal array descriptor a relatively
straightforward task.  An immediate benefit is that direct calls to
the optimized Adlib collective communication library will be possible
from the modified GA.  This work was undertaken as part of an effort
towards converging the two libraries.


\section{Relation to Other Work}
\label{sec:relation}

ARMCI differs from other systems in several key aspects.  Unlike
facilities that primarily support remote memory copies with contiguous
data (for example SHMEM, LAPI, or the Active Message run-time-system for
Split-C, which includes put/get operations \cite{Culler93}) it attempts
to optimize noncontigous data transfers, such as those involving
sections of multidimensional arrays, and generalized scatter/gather
operations.  The Cray SHMEM library \cite{Barriuso94} available on
the Cray and SGI platforms, and on clusters of PCs through HPVM
\cite{Chien97}, supports only put/get operations for contiguous
data and scatter/gather operations for the basic datatype elements.

The Virtual Interface Architecture (VIA) \cite{VIA} is a recent
network architecture and interface standard that is relevant to
ARMCI as a possible implementation target.  VIA offers support for
noncontiguous data transfers.  However, its scatter/gather Remote
Direct Memory Access (RDMA) operations can transfer data only
between noncontiguous locations in local memory and a contiguous
remote memory area.  For transfer of data between sections of local
and remote multidimensional arrays, the VIA would require an additional
copy to and from an intermediate contiguous buffer, which obviously
has an impact on performance.  This is unnecessary in ARMCI.  ARMCI
and VIA offer similar ordering on communication operations.

The MPI-2 one-sided communication specifications include remote memory
copy operations such as {\em put} and {\em get} \cite{MPI2}.
Noncontiguous data transfers are fully supported through the MPI
derived data types.  There are two models of one-sided communication
in MPI-2: ``active-target'' and ``passive-target''.  The MPI-2 one-sided
communication and in particular its ``active-target'' version have been
derived from message-passing, and its semantics include rather restrictive
(for a remote memory copy) progress rules closer to MPI-1 than to
existing remote memory copy interfaces like Cray SHMEM, IBM LAPI or
Fujitsu MPlib.  A version of MPI-2 one-sided communication called
``passive-target'' offers more relaxed progress rules and a simpler
to use model than ``active-target''.  However, it also introduces
potential performance penalties by requiring locking before access
to the remote memory (``window'') and forbidding concurrent accesses
to non-overlapping locations in a ``window''.  ARMCI does not have
similar restrictions and offers a simpler programming model that MPI-2.
This makes it more appropriate for libraries and tools supporting
some application domains, including computational chemistry.

\section{Conclusions and Future Work}

We introduced a new portable communication library targetting parallel
distributed array libraries and compiler run-time systems.  By
supporting communication interfaces for noncontiguous data transfers
ARMCI offers potential performance advantages over existing systems
for communication patterns used in many scientific applications.
We presented its performance for the strided get and gather-type
data transfers on the IBM SP where the ARMCI implementation outperformed
by a large margin the native remote memory copy operations.  ARMCI already
demonstrated its value: we employed ARMCI to implement the Global Arrays
library and it new higher-dimensional array capabilities, and used it
to optimize collective communication in the Adlib run-time system.
The porting and tuning efforts will be continued.  The future ports
will include networks of workstations and other scalable systems.
In addition, future extensions of the existing functionality will
include nonblocking interfaces to {\em get} oeprations and new
operations such as locks.

\section*{Acknowledgements}

This work was supported through the U.S. Department of Energy by
the Mathematical, Information, and Computational Science Division
the Office of Computational and Technology Research DOE 2000 ACTS
project, and used the Molecular Science Computing Facility in the
Environmental Molecular Sciences Laboratory at the Pacific Northwest
National Laboratory (PNNL).  PNNL is operated by Battelle Memorial
Institute for the U.S. Department of Energy under Contract
DE-AC06-76RLO 1830.

\bibliography{armci-paper}

\end{document}

