\documentclass[letterpaper]{article}
\usepackage{psfig}
%\documentstyle[psfig]{article}


% Margins etc
\setlength{\oddsidemargin}{0.50truein}
\setlength{\evensidemargin}{0.50truein}
\setlength{\topmargin}{-0.5truein}
\setlength{\textwidth}{5.5truein}
\setlength{\textheight}{9.0truein}

\begin{document}

\bibliographystyle{plain}
\pagestyle{plain}

\title{\bf MPJ: MPI-like Message Passing for Java} 
\author{
Bryan Carpenter$^1$, Vladimir Getov$^2$, Glenn Judd$^3$, \\
Tony Skjellum$^4$ and Geoffrey Fox$^1$ \\ \\
$^1$NPAC, Syracuse University, Syracuse, USA \\
$^2$School of Computer Science, University of Westminster, London, UK \\
$^3$Computer Science Department, Brigham Young University, Provo, USA \\
$^4$MPI Software Technology, Inc., Starkville, USA} 

\date{}

\maketitle

\begin{abstract}
Recently, there has been a lot of interest in using Java for parallel
programming.  Efforts have been hindered by lack of standard Java
parallel programming APIs.  To alleviate this problem, various groups
started projects to develop Java message passing systems modelled on
the successful Message Passing Interface (MPI).  Official MPI bindings
are currently defined only for C, Fortran, and C++, so early MPI-like
environments for Java have been divergent. This paper relates an effort
undertaken by a working group of the Java Grande Forum, seeking
a consensus on MPI-like APIs for Java, to enhance the viability
of parallel programming using Java.
\end{abstract}

\section{Introduction and background}

A likely prerequisite for parallel programming in a distributed
environment is a good message passing API.  Java comes with various
ready-made packages for communication, notably an easy-to-use interface to
BSD sockets, and the Remote Method Invocation (RMI) mechanism. Interesting
as these interfaces are, it is questionable whether parallel programmers
will find them especially convenient. Sockets and remote procedure
calls have been around for about as long as parallel computing has been
fashionable, and neither of them has been popular in that field. Both
communication models are optimized for client-server programming, whereas
the parallel computing world is mainly concerned with more ``symmetric''
communication, occurring in groups of interacting peers.

This peer-to-peer model of communication is captured in the successful 
Message Passing Interface (MPI) standard, established in 1994 
\cite{MPI-1}. MPI directly supports the Single Program Multiple Data 
(SPMD) model of parallel computing, wherein a group of processes cooperate
by executing identical program images on local data values. Reliable 
point-to-point communication is provided through a shared, group-wide 
communicator, instead of socket pairs. MPI allows numerous blocking, 
non-blocking, buffered or synchronous communication modes. It also 
provides a library of true collective operations (broadcast is the most 
trivial example). An extended standard, MPI-2~\cite{MPI-2}, allows for 
dynamic process creation and access to memory in remote processes. 

The MPI standard documents provided a language-independent 
specification as well as language-specific (C and Fortran) 
bindings~\cite{MPI-1}.  While the MPI-2 release of the standard added a 
C++ binding~\cite{MPI-2}, no Java binding has been offered or is planned
by the MPI Forum.  With the evident success of Java as a programming 
language, and its inevitable use in connection with parallel as well as 
distributed computing, the absence of a well-designed language-specific 
binding for message-passing with Java will lead to divergent, 
non-portable practices. 
Indeed MPI-like binding for Java were developed independently by several
teams.  These will be briefly reviewed in the next section.

%One Java MPI interface was produced by Getov and Mintchev
%\cite{Mintchev97,Getov98}.  In their work Java wrappers were automatically
%generated from the C MPI header.  This eased the implementation work,
%but did not lead to a fully object-oriented API.  A subset of MPI was
%implemented in the DOGMA system for Java-based parallel programming
%\cite{DOGMA}.  Dincer and Kadriy described an instrumented Java interface
%to MPI called {\em jmpi} \cite{Dincer98}.  Java implementations of
%the related PVM message-passing environment have been reported in
%\cite{Yalamanchilli98} and \cite{Ferrari98}.

Over the last three years supporters of the {\em Java Grande Forum}
\cite{JavaGrande} have been working actively to address some of the issues
involved in using Java for technical computation.  The goal of
the forum has been to develop consensus and recommendations on possible
enhancements to the Java language and associated Java standards, for
large-scale (``Grande'') applications.  Through a series of ACM-supported
workshops and conferences the forum has helped stimulate research on Java
compilers and programming environments.  The Message-Passing Working Group
of the Java Grande Forum was formed just over a year ago as a response
to the appearance of the various APIs for message-passing.  An immediate
goal was to discuss a common API for MPI-like Java libraries.  An initial
draft for a common API specification was distributed at Supercomputing
'98 \cite{JGFTR}.  Since then the working group met in San Francisco
and Syracuse, and a Birds of a Feather meeting was held at Supercomputing
'99.  Minutes of meetings are available at \cite{MPWG99A,MPWG99B}.
To avoid confusion with standards published by the original MPI Forum
the nascent API is called {\em MPJ}.


\section{Earlier work}

At the time the working group was created there were several known efforts
towards the design of early MPI-like interfaces for Java with three fully
functional but different implement\-ations---mpiJava \cite{Baker98},
JavaMPI \cite{MintchevGetov97c}, and MPIJ \cite{JCS-98}. The implementation
of mpiJava is based on the use of native methods to build a wrapper to
existing MPI library (MPICH). A comparable approach has been followed in
the development of JavaMPI, but the JavaMPI wrappers were automatically
generated by a special-purpose code generator. A large subset of MPI-like
functions called MPIJ is implemented in pure Java within the DOGMA system
for Java-based parallel programming. MPI Software Technology, Inc.
announced a commercial effort to develop
a message-passing framework and parallel support environment for Java
called JMPI.
%\cite{CrawfordIII97}. 
Some of these ``proof-of-concept''
implementations have been available since 1997 with successful ports on
clusters of workstations running Solaris, Windows NT, Irix, AIX, HP-UX,
MacOS, and Linux, as well as the IBM SP2, SGI Origin-2000, Fujitsu
AP3000, and Hitachi SR2201 parallel platforms.

\subsection{The mpiJava wrapper}

The mpiJava software \cite{Baker98} implements a Java binding for MPI
proposed late in 1997.
The API is modeled as closely as practical on the C++ binding
defined in the MPI 2.0 standard, specifically supporting the MPI 1.1
subset of that standard.  In some cases the extra runtime information
available in Java objects allows argument lists to be simplified
relative to the C++ binding.  In other cases restrictions of Java,
especially the fact that all arguments are passed by value in Java,
forces some changes to argument lists.  But in general mpiJava adheres
closely to earlier standards.

The implementation of mpiJava is through JNI wrappers to native MPI
software.  Interfacing Java to MPI is not always trivial.  We often see
low-level conflicts between the Java runtime and the interrupt
mechanisms used in MPI implementations.  The situation is improving as
JDK matures, and the mpiJava software now works reliably on top of
Solaris MPI implementations and various shared memory platforms.  A
port to Windows NT (based on WMPI) is available, and other ports are in
progress.

Other work in progess includes development of demonstrator
applications, and Java-specific extensions such as support for direct
communication of serializable objects.

\subsection{JavaMPI---automatic generation of MPI wrappers}

In principle,
the binding of existing MPI library to Java using JNI amounts to either
dynamically linking the library to the Java virtual machine, or linking
the library to the object code produced by a stand-alone Java compiler. 
Complications stem from the fact that Java data formats are in general 
different from those of C. Java implementations will have to use JNI 
which allows C functions to access Java data and perform format 
conversion if necessary. Such an interface is fairly convenient for 
writing {\em new} C code to be called from Java, but is not adequate for
linking {\em existing} native code.

Clearly an additional interface layer must be written in order to bind a
legacy library to Java. A large library like MPI has over a hundred
exported functions, therefore it is preferable to automate the creation
of the additional interface layer. The {\em Java-to-C interface
generator} (JCI) \cite{GGMS-99} takes as input a header file 
containing the C function prototypes of the native library. It outputs 
a number of files comprising the additional interface:
a file of C stub-functions;
files of Java class and native method declarations;
shell scripts for doing the compilation and linking.
The JCI tool generates a C stub-function and a Java native method
declaration for each exported function of the MPI library.
Every C stub-function takes arguments whose types correspond directly
to those of the Java native method, and converts the arguments into the
form expected by the C library function.

As the JavaMPI bindings have been generated automatically from
the C prototypes of MPI functions, they are very close to the C binding.
However, there is nothing to prevent 
from parting with the C--style binding and adopting a Java-style
object--oriented approach by grouping MPI functions into a hierarchy of
classes.

\subsection{MPIJ---MPI-like implementation in pure Java}
MPIJ is a completely Java-based implementation of MPI which runs as part
of the Distributed Object Group Metacomputing Architecture (DOGMA) 
system.  MPIJ implements a large subset of MPI-like functionality 
including all modes of point-to-point communication,
intracommunicator operations, groups, and user-defined reduction 
operations. Notable capabilities that are not yet implemented include 
process topologies, intercommunicators, and user-defined 
datatypes but these are arguably needed for legacy code only.

MPIJ communication uses native marshaling of primitive Java types.
On Win32 platforms this technique allows MPIJ to achieve communication
speeds comparable to, and in some instances exceeding, native MPI
implementations~\cite{JCSG-99}. Our performance evaluation experiments
show that Java communication speed would be greatly increased if native
marshaling were a core Java function.

A key feature of a pure Java MPI-like implementation is the ability 
to function on applet-based nodes.  In MPIJ, this provides a flexible 
method for creating clusters of workstations without the need to install
any system or user software related to the message-passing environment
on the participating nodes.


\section{The MPJ API Specification}
The MPI standard is explicitly object-based. The C and Fortran bindings
rely on ``opaque objects'' that can be manipulated only by acquiring 
object handles from constructor functions, and passing the handles to 
suitable functions in the library. The C++ binding specified in the 
MPI-2 standard collects these objects into suitable class hierarchies 
and defines most of the library functions as class member functions. 
The draft MPJ API specification follows this model, lifting the structure
of its class hierarchy directly from the C++ binding.  

The initial specification builds directly on the MPI-1 infrastructure
provided by the MPI Forum, together with language bindings motivated
by the C++ bindings of MPI-2.  The purpose of this phase of the effort
is to provide an immediate, ad hoc standardization for common message
passing programs in Java, as well as to provide a basis for conversion
between C, C++, Fortran 77, and Java.  Eventually, support for other
parts of MPI-2 also belong here, particularly dynamic process
management\footnote{Given its spartan implementation in the non-Java
space, we may not need the whole of MPI-2.}.  The position of the
working group was that the initial MPI-centric API should subsequently
be extended with more object-oriented, Java-centric features, although
the exact requirements for this later phase have not yet been established.

\begin{figure}[tbp]
\centerline{\psfig
{figure=MPJ.eps,width=4.5in}}
\caption{Principal classes of MPJ\label{hierarchy}}
\end{figure}

The major classes of the initial MPJ spec are illustrated in
Figure~\ref{hierarchy}.  The class {\tt MPJ} only has static members. It
acts as a module containing global services, such as initialization,
and many global constants including the default communicator {\tt
COMM\_WORLD}.  The most important class in the package is the communicator
class {\tt Comm}.  All communication functions in MPJ are members of {\tt
Comm} or its subclasses.  As usual in MPI, a communicator stands for a
``collective object'' logically shared by a group of processors. The
processes communicate, typically by addressing messages to their peers
through the common communicator. A class that will be important in the
following discussion is the {\tt Datatype} class. This describes the type
of the elements in the message buffers passed to send, receive, and all
other communication functions.

In general the point-to-point communication operations are
realized as methods of the {\tt Comm} class.
The basic point-to-point communication operations are {\em send} and
{\em receive}.  Their use is illustrated in Figure \ref{fig:hello}.

\begin{figure}[tbp]
\small
\begin{verbatim}
  import mpj.* ;

  class Hello {
    static public void main(String[] args) {
      MPJ.init(args) ;

      int myrank = MPJ.COMM_WORLD.rank() ;
      if(myrank == 0) {
        char [] message = "Hello, there".toCharArray() ;
        MPJ.COMM_WORLD.send(message, 0, message.length, MPJ.CHAR, 1, 99) ;
      }
      else {
        char [] message = new char [20] ;
        MPJ.COMM_WORLD.recv(message, 0, 20, MPJ.CHAR, 0, 99) ;
        System.out.println("received:" + new String(message) + ":") ;
      }

      MPJ.finish();
    }
  }
\end{verbatim}
\normalsize
\caption{Example MPJ program\label{fig:hello}}
\end{figure}

Consider the MPJ analogue of the operation {\tt MPI\_SEND}.
The method prototype is:
\begin{verbatim}
        void Comm.send(Object buf, int offset, int count,
                       Datatype datatype, int dest, int tag)
\end{verbatim}
\begin{tabular}{ll}
\hspace{0.7in} {\tt buf}      & send buffer array \\
\hspace{0.7in} {\tt offset}   & initial offset in send buffer \\
\hspace{0.7in} {\tt count}    & number of items to send \\
\hspace{0.7in} {\tt datatype} & datatype of each item in send buffer \\
\hspace{0.7in} {\tt dest}     & rank of destination \\
\hspace{0.7in} {\tt tag}      & message tag
\end{tabular}
\vspace{0.15in}
\newline
The data part of the message consists of a sequence of
{\tt count} values, each of the type indicated by {\tt datatype}.
The actual argument associated with {\tt buf} must be an array with
elements of corresponding type.  The value {\tt offset} is a subscript
in this array, defining the position of the first item of the message.

The elements of {\tt buf} may have primitive type or class type.
If the elements are objects, they must be serializable objects.  If the
{\tt datatype} argument represents an MPI basic type, its value must be
consistent with the element type of {\tt buf}.  The basic datatype values
are {\tt MPJ.BYTE}, {\tt MPJ.CHAR}, {\tt MPJ.SHORT}, {\tt MPJ.BOOLEAN},
{\tt MPJ.INT}, {\tt MPJ.LONG}, {\tt MPJ.FLOAT}, {\tt MPJ.DOUBLE}, and
{\tt MPJ.OBJECT}.  If the values is {\tt MPJ.OBJECT} the objects in
the buffer are transparently serialized and unserialized inside the
communication operations.
The {\tt datatype} argument is not redundant in the current
specification of MPJ, because the
proposal includes support for an analogue of MPI {\em derived types}.
The derived types of MPJ are restricted to have a unique {\em base
type}, one of the nine types enumerated above.  If the {\tt datatype}
argument of a communication function represents an MPJ derived type, its
base type must agree with the Java element type of the associated
{\tt buf} argument.  

We note that if it was decided to remove derived types from
MPJ, {\tt datatype} arguments could be removed from many functions, and
Java runtime inquiries could be used internally to extract the element
type of the buffer\footnote{Or methods like {\tt send} could be overloaded
to accept buffers with elements of the 9 basic types.  The disadvantage of
this is that it leads to a major proliferation in the number of methods.}.


\subsection{MPJ as an MPI Language Binding\label{binding}}

MPJ does not have the status of an official language binding for MPI.
But, as a matter of interest, this section will compare some surface
features of the Java API with standard MPI language bindings.

All MPJ classes belong to the package {\tt mpj}.
Conventions for capitalization, etc, in class and member names
generally follow the recommendations of Sun's Java code conventions
\cite{codeconv}.  
In general these conventions are consistent with the naming conventions
of the MPI 2.0 C++ standard.  Exceptions to this rule include the
use of lower case for the first letters of method names,
and avoidance of underscore in variable names.

With MPI opaque objects replaced by Java objects, MPI destructors
can be absorbed into Java object destructors ({\tt finalize} methods),
called automatically by the Java garbage collector.  MPJ adopts this
strategy as the general rule.  Explicit calls to destructor functions
are typically omitted from the Java user code.  An exception is made
for the {\tt Comm} classes.  In MPI the destructor for a communicator is
a collective operation, and the user must ensure that calls are made at
consistent times on all processors involved.  Automatic garbage collection
would not guarantee this.  Hence the MPJ {\tt Comm} class has an explicit
{\tt free} method.

Some options allowed for {\em derived data types} in the C and Fortran
binding are absent from MPJ.  The Java VM does not support any concept of
a global linear address space.  Physical memory displacements between
fields in objects are unavailable or ill-defined.  This puts some limits
on the possible uses of any analogue of the {\em MPI\_TYPE\_STRUCT}
type constructor.  In practise the MPJ {\tt struct} datatype constructor
has been further restricted in a way that makes it impossible to send
mixed {\em basic datatypes} in a single message.  Since the set of basic
datatypes recognized is extended to include serializable Java
{\em objects}, this should not be a serious problem.

Array size arguments are often omitted in MPJ, because they can be picked
up within the function by reading the {\tt length} member of the array
argument.  A crucial exception is for message buffers, where an explicit
count is always given.  Message buffers aside, typical array arguments
to MPI functions (e.g., vectors of request structures) are small arrays.
If subsections of these must be passed to an MPI function, the sections
can be copied to smaller arrays at little cost.  In contrast, message
buffers are typically large and copying them is expensive, so it is
worthwhile to pass an extra size argument to select a subset. (Moreover,
if derived data types are being used, the required value of the {\tt
count} argument is always different to the buffer length.)  C and
Fortran both have ways of treating a section of an array, offset from
the beginning of the array, as if it was an array in its own right.
Java does not have any such mechanism.  To provide the same flexibility
in MPJ, an explicit integer {\tt offset} parameter also accompanies any
buffer argument.  This defines the position in the Java array of the
first element actually treated as part of the buffer.

The C and Fortran languages define a straightforward mapping (or
``sequence association'') between their multidimensional arrays
and equivalent one-dimensional arrays.  In MPI a multidimensional
array passed as a message buffer argument is generally treated like
a one-dimensional array with the same element type.  Offsets in the
buffer (such as offsets occuring in derived data types) behave like
offsets in the effective one-dimensional array.  In Java the relationship
between multidimensional arrays and one dimensional arrays is different.
An ``$n$-dimensional array'' is equivalent to a one-dimensional array of
$(n - 1)$-dimensional arrays.  In the MPJ interface message buffers are
always treated as one-dimensional arrays.  The element type {\em may}
be an object, which {\em may} have array type.  Hence multidimensional
arrays can appear as message buffers, but the interpretation and behaviour
is significantly different.

Unlike the standard MPI interfaces, MPJ methods do not return explicit
error codes.  Instead, the Java exception mechanism is used to
report errors.

\subsection{Complete Draft API}

The appendix of this paper lists the public interfaces of all the classes.
Of course this only defines syntax.  A more complete description of the
semantics of all methods is available in \cite{JGFTR}.

\section{Open Issues}

\label{sec:open}

The API described in \cite{JGFTR} is not assumed to be ``final''.
It was originally presented as a starting point for discussion.  In this
section we will mention some areas we consider to be open
to improvement.

\subsection{Derived datatypes.}

It is unclear whether a Java interface should support MPI-like derived 
data types.  A proposal for a Java-compatible subset of derived types
is included in the draft specification document~\cite{JGFTR}, but 
deleting it would simplify the API significantly.  In particular
{\tt datatype} arguments for buffers could be dropped.

One factor in favor of including MPI-like derived data types in MPJ is
the support for legacy MPI applications.  The possible need to interact
with native code that uses derived datatypes is probably best supported
by including derived datatypes in the MPJ API specification.

It has been argued that the functionality of derived datatypes is
already provided by Java objects, and supporting both only adds unneeded
complexity.  But in fact there are good reasons to retain some additional
functionality of derived datatypes.  Any scientific code, written in
Java or otherwise, will benefit from the ability to efficiently and
conveniently send {\em sections} (subsets) of program arrays.  In MPI,
this is one of the most useful roles of the so-called derived datatypes,
and MPJ object datatypes do not address this requirement.
The discussion of whether derived datatypes are to be supported in MPJ
should therefore be closely linked with the discussion of how true
``scientific'' (multi-dimensional) arrays, allowing Fortran-90-like
sectioning operations, should be handled.

\subsection{Multidimensional arrays}

Some specific support for communicating
multidimensional arrays would be desirable.  In the current proposal,
sending a multidimensional array involves either sending one row
at a time or using Java object serialization, both of which will introduce
performance bottlenecks.  For instance, our experience has shown that
MPIJ sends a 200x200 array of doubles over Fast Ethernet much faster
when multidimensional array support is included than when individual
rows are sent.  More detailed analysis of this problem is presented
in \cite{Carpenter99}.

Trying to fix the problem for standard Java multidimensional arrays is
probably the wrong approach.  There is a deeper problem that the Java
``array-of-arrays'' model for multidimensional arrays is not especially
well-suited for ``scientific'' computation.  This issue is being actively
addressed by other groups in the Java Grande Forum.  In particular the
work by IBM on the Array package \cite{Moreira99}, which has been
adopted by the Java Grande numerics working group, is very relevant.
A more complete MPJ specification should probably include mechanisms
for efficiently communicating standardized ``scientific'' arrays, and
their sections.  

In fact, if a standard like the Array package were adopted, and if it
supported description of array sections (without copying elements), it
is quite likely that the remaining arguments in favour of keeping an
MPI-like derived datatype mechanism would go away.

\subsection{Overloaded communication operations}

It has been suggested that many of the communication operations
should be overloaded to provide simplified variants that omit arguments
like {\tt offset}, {\tt count} (and possibly {\tt datatype}).
This suggestion is not included in the current proposal, but it could
be added.  The primary argument in
favor is that it simplifies user code.  For instance,
\begin{verbatim}
    MPI.COMM_WORLD.send(message, 0, message.length, MPI.CHAR, 1, 99);
\end{verbatim}
becomes
\begin{verbatim}
    MPI.COMM_WORLD.send(message, MPI.CHAR, 1, 99);
\end{verbatim}
The obvious counter-argument is that this very significantly increases
the total number of methods in the API.
A possible compromise is to provide overloaded versions only of specific
common functions such as point-to-point communication functions
(the argument against this, in turn, is that it looks inconsistent).


\subsection{Other issues}

The current MPJ spec has all MPI-like exceptions derived from {\tt
MPJExc\-eption}.  An alternative suggestion that has been put forward is that all
MPJ exceptions be derived from two classes: {\tt MPJException} and {\tt
MPJRuntimeException}.  Subclasses of {\tt MPJExc\-eption} would represent
errors that the user would be required to catch whereas subclasses of
{\tt MPJRuntimeException} would represent uncommon or unusual errors.
It has also been suggested that certain MPJ exceptions could carry
subexceptions when the cause of the MPJ exception is another exception.
The use of MPI-like user-defined and predefined error handlers is another
an open question.  These could still serve a purpose in addition to the
exception mechanism mentioned earlier.  
%For instance, a predefined
%error handler {\em MPJ\_ERRORS\_ARE\_FATAL} could call {\tt MPJ.abort},
%while a predefined error handler {\em MPJ\_ERRORS\_RETURN} would leave
%control over when {\tt MPJ.abort} was called to the user.

It has been suggested that the specification of user defined operations
could be modified.  In the current proposal users create a {\tt
UserDefinedOperation} and use this to create an {\tt Op}.  This results
in the creation of a class ({\tt UserDefinedOperation}) which is not
really necessary.  An alternative would be to simply have users define
subclasses of {\tt Op} with method named, say, {\tt call}. This design
would also eliminate the overhead of a method invocation.

A profiling interface for MPJ has not yet been defined.  A 
possible general design approach is for profiling class and
method names to exactly match those of the non-profiling
classes and methods.  Implementors would then place the compiled
binary files in different locations. As Java linking is 
always dynamic, this would allow users to enable or disable 
profiling simply selecting the appropriate codebase  
(e.g. by changing the CLASSPATH environment variable).

%\section{Related and Future Work}

\section{Discussion}

An initial goal of the Java Grande Message Passing working group was to
promote a standardized MPI binding for Java.  It became apparent that
this road was likely to produce a collision of interest with the existing
MPI community, and the name of the new API was changed
to MPJ.  MPJ was designated an ``MPI-like'' specification.  The current
specification is available in \cite{JGFTR}.  This specification is
essentially complete and self-contained, but as discussed in section
\ref{sec:open}, it is not necessarily considered ``final''.

Because it was designed on object-oriented principles, most of the MPI
specification actually maps very naturally into Java.  So long as one
accepts the Java Grande premise that Java is an excellent basis for
technical computing, an MPI-like approach to parallel computing seems
very promising---more promising than some have assumed.  But there
remain non-obvious issues about supporting basic MPI functionality.
Some of the more difficult ones boil down to the lack of a good model of
scientific arrays in Java.  This issue is somewhat outside the purview
of this working group, but is being actively discussed by the Java Grande
Numerics Working Group \cite{JavaNumerics}.

Reference implementations of the specification are under development.
An implementation based on JNI wrappers to native MPI will be
created by adapting the mpiJava wrappers \cite{Baker98}.
While this is a good approach in some situations, it
has various disadvantages and conflicts with the ethos of Java, where
pure-Java, write-once-run-anywhere software is the order of the day.
We outlined in \cite{Baker00} a design for a {\em pure-Java}
reference implementation.  Design goals were that the system should be as
easy to install on distributed systems as we can reasonably make it, and
that it be sufficiently robust to be useable in an Internet environment.

MPI 1 was designed with relatively static platforms in mind.
To better support computing in volatile Internet environments, we will
need (at least) features from MPI 2 like dynamic spawning of process groups
and parallel client/server interfaces.  We note that a natural framework
for dynamically discovering new compute resources and establishing
connections between running programs already exists in Sun's Jini project
\cite{Arnold99}, and one line of investigation is into MPJ implementations
operating in the Jini framework.

%\begin{itemize}
%\item Intelligent generator of wrappers to legacy MPI libraries
%\item Test suite
%\item Java-MPI Benchmarks
%\end{itemize}

Closely modelled as it is on the MPI standards, the existing MPJ
specification should be regarded as a first phase in a broader program to
define a more Java-centric high performance message-passing environment.
In future a detachment from legacy implementations involving Java on
top of native methods will be emphasized.  We should consider
the possibility of layering the messaging middleware over standard
transports and other Java-compliant middleware (like CORBA). In a sense,
the middleware developed at this level should offer a choice of emphasis
between performance or generality, while always supporting portability.
We note an opportunity to study and standardize aspects of real-time
and fault-aware programs, drawing on the concepts learned in the {\em
MPI/RT} activity \cite{Kanevsky98}.  For performance, we should seek to take advantage of
what has been learned since MPI-1 and MPI-2 were finalized, or ignored in
MPI standardization for various reasons---for instance drawing on the body
of knowledge completed within the MPI/RT Forum.  From here we may at least
glean design hints concerning channel abstractions, and the more direct
use of object-oriented design for message passing than was seen in MPI-1
or MPI-2.  The value of this type of messaging middleware in the
embedded and real-time Java application spaces should also be considered.

We should aim to offer MPI-like services to Java programs in an
upward compatible fashion.  The purposes are twofold: performance and
portability.

\appendix

\section{Public Interface of Classes in MPJ Draft Specification}

\subsection{\tt MPJ}

\footnotesize
\begin{verbatim}
public class MPJ {
  public static Intracomm COMM_WORLD;

  public static Datatype BYTE, CHAR, SHORT, BOOLEAN, INT, LONG,
                         FLOAT, DOUBLE, OBJECT, PACKED, LB, UB ;

  public static int ANY_SOURCE, ANY_TAG ;

  public static int PROC_NULL ;

  public static int BSEND_OVERHEAD ;

  public static int UNDEFINED ;

  public static Op MAX, MIN, SUM, PROD, LAND, BAND,
                   LOR, BOR, LXOR, BXOR, MINLOC, MAXLOC ;

  public static Datatype SHORT2, INT2, LONG2, FLOAT2, DOUBLE2 ;

  public static Group GROUP_EMPTY ;

  public static Comm COMM_SELF ;

  public static int IDENT, CONGRUENT, SIMILAR, UNEQUAL ;

  public static int GRAPH, CART ;

  public static ErrHandler ERRORS_ARE_FATAL, ERRORS_RETURN ;

  public static int TAG_UB, HOST, IO ;


  // Buffer allocation and usage

  public static void bufferAttach(byte [] buffer) throws MPJException {...}

  public static byte [] bufferDetach() throws MPJException {...}


  // Environmental Management

  public static String [] init(String[] argv) throws MPJException {...}

  public static void finish() throws MPJException {...}

  public static String getProcessorName() throws MPJException {...}

  public static double wtime() throws MPJException {...}

  public static double wtick() throws MPJException {...}

  public static boolean initialized() throws MPJException {...}

  ...
}
\end{verbatim}
\normalsize

\subsection{\tt Comm}

\footnotesize
\begin{verbatim}
public class Comm {

  // Communicator Management

  public int size() throws MPJException {...}

  public int rank() throws MPJException {...}

  public Group group() throws MPJException {...}  // (section "Group management" of spec)

  public static int compare(Comm comm1, Comm comm2) throws MPJException {...}

  public Object clone() {...}

  public void free() throws MPJException {...}


  // Inter-communication

  public boolean testInter() throws MPJException {...}

  public Intercomm createIntercomm(Comm localComm, int localLeader,
                                   int remoteLeader, int tag) throws MPJException {...}


  // Caching

  public Object attrGet(int keyval) throws MPJException {...}


  // Blocking Send and Receive operations

  public void send(Object buf, int offset, int count, 
                   Datatype datatype, int dest, int tag) throws MPJException {...}

  public Status recv(Object buf, int offset, int count, 
                     Datatype datatype, int source, int tag) throws MPJException {...}


  // Communication Modes

  public void bsend(Object buf, int offset, int count, 
                    Datatype datatype, int dest, int tag) throws MPJException {...}

  public void ssend(Object buf, int offset, int count, 
                    Datatype datatype, int dest, int tag) throws MPJException {...}

  public void rsend(Object buf, int offset, int count, 
                    Datatype datatype, int dest, int tag) throws MPJException {...}


  // Nonblocking communication

  public Request isend(Object buf, int offset, int count, 
                       Datatype datatype, int dest, int tag) throws MPJException {...}

  public Request ibsend(Object buf, int offset, int count, 
                        Datatype datatype, int dest, int tag) throws MPJException {...}

  public Request issend(Object buf, int offset, int count, 
                        Datatype datatype, int dest, int tag) throws MPJException {...}

  public Request irsend(Object buf, int offset, int count, 
                        Datatype datatype, int dest, int tag) throws MPJException {...}

  public Request irecv(Object buf, int offset, int count, 
                       Datatype datatype, int source, int tag) throws MPJException {...}


  // Probe and cancel

  public Status iprobe(int source, int tag) throws MPJException {...}

  public Status probe(int source, int tag) throws MPJException {...}


  // Persistent communication requests

  public Prequest sendInit(Object buf, int offset, int count, 
                           Datatype datatype, int dest, int tag) throws MPJException {...}

  public Prequest bsendInit(Object buf, int offset, int count, 
                            Datatype datatype, int dest, int tag) throws MPJException {...}

  public Prequest ssendInit(Object buf, int offset, int count, 
                            Datatype datatype, int dest, int tag) throws MPJException {...}

  public Prequest rsendInit(Object buf, int offset, int count, 
                            Datatype datatype, int dest, int tag) throws MPJException {...}

  public Prequest recvInit(Object buf, int offset, int count, 
                           Datatype datatype, int source, int tag) throws MPJException {...}


  // Send-receive

  public Status sendrecv(Object sendbuf, int sendoffset, int sendcount, Datatype sendtype,
                         int dest, int sendtag,
                         Object recvbuf, int recvoffset, int recvcount, Datatype recvtype,
                         int source, int recvtag) throws MPJException {...}

  public Status sendrecvReplace(Object buf, int offset, int count, Datatype datatype,
                                int dest, int sendtag,
                                int source, int recvtag) throws MPJException {...}

  // Pack and unpack

  public int pack(Object inbuf, int offset, int incount, Datatype datatype,
                  byte [] outbuf, int position) throws MPJException {...}

  byte[] pack(Object inbuf, int offset, int incount, Datatype datatype)
                      throws MPJException {...}

  public int unpack(byte [] inbuf, int position,
                    Object outbuf, int offset, int outcount, Datatype datatype)
                            throws MPJException {...}

  public int packSize(int incount, Datatype datatype) throws MPJException {...}


  // Process Topologies

  int topoTest() throws MPJException {...}


  // Environmental Management

  public static void errorhandlerSet(Errhandler errhandler) throws MPJException {...}

  public static Errhandler errorhandlerGet() throws MPJException {...}

  void abort(int errorcode) throws MPJException {...}

  ...
}
\end{verbatim}
\normalsize


\subsection{{\tt Intracomm} and {\tt Intercomm}}

\footnotesize
\begin{verbatim}
public class Intracomm extends Comm {

  public Object clone() { ... }

  public Intracomm create(Group group) throws MPJException {...}

  public Intracomm split(int colour, int key) throws MPJException {...}

  // Collective communication

  public void barrier() throws MPJException {...}

  public void bcast(Object buffer, int offset, int count,
                    Datatype datatype, int root) throws MPJException {...}

  public void gather(Object sendbuf, int sendoffset, int sendcount, Datatype sendtype,
                     Object recvbuf, int recvoffset, int recvcount, Datatype recvtype,
                     int root) throws MPJException {...}

  public void gatherv(Object sendbuf, int sendoffset, int sendcount, Datatype sendtype,
                      Object recvbuf, int recvoffset, int [] recvcount, int [] displs,
                      Datatype recvtype, int root) throws MPJException {...}

  public void scatter(Object sendbuf, int sendoffset, int sendcount, Datatype sendtype,
                      Object recvbuf, int recvoffset, int recvcount, Datatype recvtype,
                      int root) throws MPJException {...}

  public void scatterv(Object sendbuf, int sendoffset, int [] sendcount, int [] displs,
                       Datatype sendtype,
                       Object recvbuf, int recvoffset, int recvcount, Datatype recvtype,
                       int root) throws MPJException {...}

  public void allgather(Object sendbuf, int sendoffset, int sendcount, Datatype sendtype,
                        Object recvbuf, int recvoffset, int recvcount, Datatype recvtype)
                                throws MPJException {...}

  public void allgatherv(Object sendbuf, int sendoffset, int sendcount, Datatype sendtype,
                         Object recvbuf, int recvoffset, int [] recvcounts, int [] displs,
                         Datatype recvtype) throws MPJException {...}

  public void alltoall(Object sendbuf, int sendoffset, int sendcount, Datatype sendtype,
                       Object recvbuf, int recvoffset, int recvcount, Datatype recvtype)
                               throws MPJException {...}

  public void alltoallv(Object sendbuf, int sendoffset, int [] sendcount, int [] sdispls,
                        Datatype sendtype,
                        Object recvbuf, int recvoffset, int [] recvcount, int [] rdispls,
                        Datatype recvtype) throws MPJException {...}

  public void reduce(Object sendbuf, int sendoffset, Object recvbuf, int recvoffset,
                     int count, Datatype datatype, Op op, int root) throws MPJException {...}

  public void allreduce(Object sendbuf, int sendoffset, Object recvbuf, int recvoffset,
                        int count, Datatype datatype, Op op) throws MPJException {...}

  public void reduceScatter(Object sendbuf, int sendoffset,
                            Object recvbuf, int recvoffset,
                            int [] recvcounts, Datatype datatype,
                            Op op) throws MPJException {...}

  public void scan(Object sendbuf, int sendoffset, Object recvbuf, int recvoffset,
                   int count, Datatype datatype, Op op) throws MPJException {...}


  // Topology Constructors

  public Graphcomm createGraph(int [] index, int [] edges,
                               boolean reorder) throws MPJException {...}

  public Cartcomm createCart(int [] dims, boolean [] periods,
                             boolean reorder) throws MPJException {...}

  ...
}
\end{verbatim}
\normalsize

\footnotesize
\begin{verbatim}
public class Intercomm extends Comm {

  public Object clone() { ... }

  // Inter-communication

  public int remoteSize() throws MPJException {...}

  public Group remoteGroup() throws MPJException {...}

  public Intracomm merge(boolean high) throws MPJException {...}

  ...
}
\end{verbatim}
\normalsize

\subsection{\tt Op}

\footnotesize
\begin{verbatim}
public class Op {
  Op(UserFunction function, boolean commute) throws MPJException {...}

  void finalize() {...}

  ...
}
\end{verbatim}
\normalsize

\subsection{\tt Group}

\footnotesize
\begin{verbatim}
public class Group {

  // Group Management

  public int size() throws MPJException {...}

  public int rank() throws MPJException {...}

  public int [] translateRanks(Group group1, int [] ranks1) throws MPJException {...}

  public static int compare(Group group1, Group group2) throws MPJException {...}

  public static Group union(Group group1, Group group2) throws MPJException {...}

  public static Group intersection(Group group1, Group group2) throws MPJException {...}

  public static Group difference(Group group1, Group group2) throws MPJException {...}

  public Group incl(int [] ranks) throws MPJException {...}

  public Group excl(int [] ranks) throws MPJException {...}

  public Group rangeIncl(int [] [] ranges) throws MPJException {...}

  public Group rangeExcl(int [] [] ranges) throws MPJException {...}

  public void finalize() {...}

  ...
}
\end{verbatim}
\normalsize

\subsection{\tt Status}

\footnotesize
\begin{verbatim}
public class Status {

  public int index ;


  // Blocking Send and Receive operations

  public int getCount(Datatype datatype) throws MPJException {...}

  public int getSource() throws MPJException {...}
  public int getTag() throws MPJException {...}

  // Nonblocking communication

  public int getIndex() throws MPJException {...}

  // Probe and Cancel

  public boolean testCancelled() throws MPJException {...}


  // Derived datatypes

  public int getElements(Datatype datatype) throws MPJException {...}

  ...
}
\end{verbatim}
\normalsize

\subsection{{\tt Request} and {\tt Prequest}}

\footnotesize
\begin{verbatim}
public class Request {

  // Nonblocking communication

  public Status wait() throws MPJException {...}

  public Status test() throws MPJException {...}

  public Request() throws MPJException {...}

  public void finalize() {...}

  public boolean isVoid() throws MPJException {...}

  public static Status waitAny(Request [] arrayOfRequests) throws MPJException {...}

  public static Status testAny(Request [] arrayOfRequests) throws MPJException {...}

  public static Status [] waitAll(Request [] arrayOfRequests) throws MPJException {...}

  public static Status [] testAll(Request [] arrayOfRequests) throws MPJException {...}

  public static Status [] waitSome(Request [] arrayOfRequests) throws MPJException {...}

  public static Status [] testSome(Request [] arrayOfRequests) throws MPJException {...}


  // Probe and cancel

  public void cancel() throws MPJException {...}

  ...
}
\end{verbatim}
\normalsize

\footnotesize
\begin{verbatim}
public class Prequest extends Request {

  // Persistent communication requests

  public void start() throws MPJException {...}

  public static void startAll(Request [] arrayOfRequests) throws MPJException {...}

  ...
}
\end{verbatim}
\normalsize

\subsection{\tt Datatype}

\footnotesize
\begin{verbatim}
public class Datatype {

  // Derived datatypes

  public Datatype contiguous(int count) throws MPJException {...}

  public Datatype vector(int count, int blocklength, int stride) throws MPJException {...}

  public Datatype hvector(int count, int blocklength, int stride) throws MPJException {...}

  public Datatype indexed(int [] arrayOfBlocklengths,
                          int [] arrayOfDisplacements) throws MPJException {...}

  public Datatype hindexed(int [] arrayOfBlocklengths,
                           int [] arrayOfDisplacements) throws MPJException {...}

  public static Datatype struct(int [] arrayOfBlocklengths,
                                int [] arrayOfDisplacements,
                                Datatype [] arrayOfTypes) throws MPJException {...}

  public int extent() throws MPJException {...}

  public int size() throws MPJException {...}

  public int lb() throws MPJException {...}

  public int ub() throws MPJException {...}

  public void commit() throws MPJException {...}

  public void finalize() {...}

  ...
}
\end{verbatim}
\normalsize

\subsection{Classes for virtual topologies}

\footnotesize
\begin{verbatim}
public class Cartcomm extends Intracomm {

  public Object clone() { ... }

  // Topology Constructors

  static public dimsCreate(int nnodes, int [] dims) throws MPJException {...}

  public CartParms get() throws MPJException {...}

  public int rank(int [] coords) throws MPJException {...}

  public int [] coords(int rank) throws MPJException {...}

  public ShiftParms shift(int direction, int disp) throws MPJException {...}

  public Cartcomm sub(boolean [] remainDims) throws MPJException {...}

  public int map(int [] dims, boolean [] periods) throws MPJException {...}
}
\end{verbatim}
\normalsize

\footnotesize
\begin{verbatim}
public class CartParms {

  // Return type for Cartcomm.get()

  public int [] dims ;
  public booleans [] periods ;
  public int [] coords ; 
}
\end{verbatim}
\normalsize

\footnotesize
\begin{verbatim}
public class ShiftParms {

  // Return type for Cartcomm.shift()

  public int rankSource ;
  public int rankDest ; 
}
\end{verbatim}
\normalsize


\footnotesize
\begin{verbatim}
public class Graphcomm extends Intracomm {

  public Object clone() {...}

  // Topology Constructors

  public GraphParms get() throws MPIException {...}

  public int [] neighbours(int rank) throws MPIException {...}

  public int map(int [] index, int [] edges) throws MPIException {...}
}
\end{verbatim}
\normalsize

\footnotesize
\begin{verbatim}
public class GraphParms {

  // Return type for Graphcomm.get()

  public int [] index ;
  public int [] edges ; 
}
\end{verbatim}
\normalsize

\bibliography{cpempi}

\end{document}

