\chapter{mpiJava: An Object-Oriented Java Interface to MPI}
\label{chp:mpiJava}

\pagestyle{myheadings}
\markright{CHAPTER\ref{chp:mpiJava}.  MPIJAVA: A JAVA INTERFACE TO MPI}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In this chapter, we turn to the main subject of this dissertation,
 {\em mpiJava}---a Java interface to the widely used Message Passing
 Interface (MPI).

In the first part of this chapter, we discuss the design of the
mpiJava API and issues associated with its development.  In
the second part of this chapter, we briefly outline the steps
necessary to ``port'' mpiJava onto a range of operating
systems, including Windows NT, Linux and Solaris.  In the final part
of this chapter, we present and then discuss some performance
measurements made of communications bandwidth and latency to compare
mpiJava on these systems.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Motivation}

Developers of parallel applications generally use the Single Program
Multiple Data (SPMD) model of parallel computing, wherein a group of
processes cooperate by executing identical program images on local
data values.  A programmer using the SPMD model has a choice of
explicit or implicit  means to move data between the cooperating
processes. Today, the normal explicit means is via message passing and
the a typical implicit means might be via data-parallel languages, such as
HPF. Although using implicit means to develop parallel applications is
generally thought to be easier, explicit message passing is more often
used since currently developers can produce more efficient and
effective parallel applications using message passing. 

A basic prerequisite for message passing is a good communication API.
Java comes with various ready-made packages for communication, notably
an interface to BSD sockets, and the Remote Method Invocation (RMI)
mechanism.  Both these communication models are optimized for
client-server programming, whereas the parallel computing world is
mainly concerned with ``symmetric'' communication, occurring in groups
of interacting peers. 

This symmetric model of communication is captured in the successful
Message Passing Interface standard (MPI), established a few years
ago \cite{MPIFORUM}. 
MPI directly supports SPMD model of parallel computing. Reliable
point-to-point communication is provided through a shared, group-wide
{\em communicator}, instead of socket pairs. MPI allows numerous blocking,
non-blocking, buffered or synchronous communication modes. It also
provides a library of true collective operations (broadcast is the
most trivial example). An extended standard, MPI 2, allows for dynamic
process creation and access to memory in remote processes.

The existing MPI standards specify language bindings for Fortran, C
and C++. In this chapter we discuss a binding of MPI 1.1 for Java, and
describe an implementation using Java wrappers to invoke C MPI calls
through the Java Native Interface(JNI) \cite{LIANG,GORDON}. 
The software was originally publically available from: 
\begin{verbatim}
  http://www.npac.syr.edu/projects/pcrc/mpiJava
\end{verbatim}
It has moved to 
\begin{verbatim}
  http://aspen.csit.fsu.edu/pss/HPJava/mpiJava.html
\end{verbatim}

The releases of mpiJava include complete source, makefiles,
configuration scripts, compiled libraries for WMPI, release test codes
(based on the IBM MPI test suite), example applications, javadoc
documentation, and installation and usage notes.  



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction to the mpiJava API\label{sec:introAPI}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The MPI standard is explicitly object-based. The C and Fortran
bindings rely on ``opaque objects'' that can be manipulated only by
acquiring object handles from constructor functions, and passing the
handles to suitable functions in the library. The C++ binding
specified in the MPI 2 standard collects these objects into suitable
class hierarchies and defines most of the library functions as class
member functions. The mpiJava API follows this model, lifting
the structure of its class hierarchy directly from the C++
binding. The major classes of mpiJava are illustrated in
Figure \ref{hierarchy}.

\begin{figure}[bt]
\centerline{\epsfig{figure=Figs/mpiJava.eps,width=5in}}
\caption{Principal classes of mpiJava\label{hierarchy}}
\end{figure}


The class MPI only has static members. It acts as a module containing
global services, such as initialization of MPI, and many global
constants including the default communicator \texttt{COMM\_WORLD}. 

The most important class in the package is the communicator class
\texttt{Comm}. All communication functions in mpiJava are
members of \texttt{Comm} or its subclasses. As usual in MPI, a
communicator stands for a ``collective object'' logically shared by a
group of processors. The processes communicate, typically by addressing
messages to their peers through the common communicator. 

Another class that is important for the discussion below is the 
\texttt{Datatype} class. This describes the type of the elements in the
message buffers passed to send, receive, and all other communication
functions. Various basic datatypes are predefined in the package. These
mainly correspond to the primitive types of Java, shown in Table
\ref{basicdatatype}. The {\tt MPI.OBJECT} is a new predefined datatype 
for derived data types.

\begin{table}[t]
\centerline{
\begin{tabular}{| l | l |} \hline
MPI datatype & Java datatype \\
\hline
MPI.BYTE     & {\tt byte} \\
MPI.CHAR     & {\tt char} \\
MPI.SHORT    & {\tt short} \\
MPI.BOOLEAN  & {\tt boolean} \\
MPI.INT      & {\tt int} \\
MPI.LONG     & {\tt long} \\
MPI.FLOAT    & {\tt float} \\
MPI.DOUBLE   & {\tt double} \\
MPI.OBJECT   & {\tt Object} \\
\hline
\end{tabular}
}
\caption{Basic datatypes of mpiJava\label{basicdatatype}}
\end{table}

The standard send and receive operations of MPI are members of
\texttt{Comm} with interfaces:
%\linebreak

\begin{verbatim}
public void Send(Object buf, int offset, int count, Datatype
                 datatype, int dest, int tag)

public Status Recv(Object buf, int offset, int count, Datatype
                   datatype, int source, int tag)
\end{verbatim}

In both cases the actual argument corresponding to \texttt{buf} must
be a Java array. In the current implementation they must be arrays
with elements of primitive type. By implication they must be
one-dimensional arrays, because Java ``multidimensional arrays'' are
really arrays of arrays. In these and all other mpiJava calls, the
buffer array argument is followed by an offset that specifies the
element of in array where the message actually starts.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{figure}[t]
\small
\Line 
\begin{verbatim}

// Simple program

import mpi.*; 

class Hello { 
  static public void main(String[] args){
    MPI.Init(args); 
    int myrank = MPI.COMM_WORLD.Rank(); 
    if(myrank == 0)}
      char [] message = "Hello, there".toCharArray() ; 
      MPI.COMM_WORLD.Send(message,0,message.length, MPI.CHAR, 1, 99); 
    } else {
      char [] message = new char [20] ; 
      MPI.COMM_WORLD.Recv(message, 0, 20, MPI.CHAR, 0, 99) ; 
      System.out.println("received:" + new String(message) + ":"); 

    }
    MPI.Finalize(); 
  }
}

\end{verbatim}
\normalsize
\Line 
\caption{Minimal mpiJava program (run in two processes)\label{hello}}
\end{figure}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Language Binding\label{binding}}

A simple ``HelloWorld'' mpiJava program is shown in figure ~\ref{hello}. 
The mpiJava API is modelled as closely as practical on the C++
binding defined in the MPI 2.0 standard (currently we only support the
MPI 1.1 subset).  A number of changes to argument lists are forced by
of the restriction that arguments cannot be passed by reference in
Java. In general outputs of mpiJava methods come through the
result value of the function.

In this section we discuss the issues and special features of Java
binding. 
As a result of the following changes mpiJava argument lists are often more
concise than the corresponding C or C++ argument lists. 

\subsubsection{Naming Conventions\label{naming}}
All MPI classes belong to the package {\tt mpi}.
Conventions for capitalization, etc, in class and member names
generally follow the C++ MPI bindings.

\subsubsection{Data types}
Opaque objects are presented as Java objects.
This introduces the
option of simplifying the user's task in managing these objects.  MPI
destructors can be absorbed into Java object destructors, which are
called automatically by the Java garbage collector.  We adopt this
strategy as the general rule.  Explicit calls to MPI destructor
functions are typically omitted from the Java user interface (they are
absorbed into \texttt{finalize} methods).  Exceptions are made for the
\texttt{Comm} and \texttt{\tt Request} classes.
\texttt{MPI\_COMM\_FREE} is a collective operation, so the user must
ensure that calls are made at consistent times by all processors
involved---the call can't be left to the vagaries of the garbage
collector.  A similar  case can be made for \texttt{MPI\_REQUEST\_FREE}.

\subsubsection{Restrictions on {\em struct} Derived Type.}
Some options allowed for {\em derived data types} in the C and Fortran
binding are deleted in mpiJava.  The Java VM does not
incorporate a concept of a global linear address space.  Passing physical
addresses to data type definitions is not allowed.  The use of the {\tt
MPI\_TYPE\_STRUCT} datatype constructor is also restricted in a way that
makes it impossible to send mixed {\em basic datatypes} in a single
message.  Since, however, the set of basic datatypes recognized by MPI
is extended to include serializable Java {\em objects}, this
should not be a serious restriction in practice. 
We will discuss derived type in detail in Section \ref{DATATYPE}.

\subsubsection{Multidimensional Arrays and Offsets.}
The C and Fortran languages define a straightforward mapping
(or ``sequence association'') between their multidimensional
arrays and equivalent one-dimensional arrays.  So in C or Fortran a
multidimensional array passed as a message buffer argument is first
interpreted as a one-dimensional array with the same element type as
the original multidimensional array.  Offsets in the buffer (such as
offsets occurring in derived data types) are then interpreted in terms of
the effective one-dimensional array (or---equivalent up to a constant
factor---in terms of physical memory).  In Java the relationship between
multidimensional arrays and one dimensional arrays is different.
An ``$n$-dimensional array'' is equivalent to a one-dimensional array
of $(n - 1)$-dimensional arrays.  In mpiJava,
message buffers are always one-dimensional arrays.
The element type {\em may} be an object, which {\em may} have array type.
Hence multidimensional arrays can appear as message buffers, but the
interpretation is subtly different.  In distinction to the C or Fortran
case {\em offsets in multidimensional message buffers are always
interpreted as offsets in the outermost one-dimensional array}.

\subsubsection{Start of Message Buffer.}
C and Fortran both have devices for treating a section of an array,
offset from the beginning of the array, as if it was an array in its
own right.  Java doesn't have any such mechanism.  To provide the same
flexibility, an {\tt offset} parameter is associated with any buffer
argument.  This defines the position of the first actual buffer element
in the Java array.

\subsubsection{Error Codes.}
Unlike the standard C and Fortran interfaces, the mpiJava interfaces to MPI
calls do not return explicit error codes.  The Java exception
mechanism will be used to report errors.
The exception mechanism is very widely
used by Java libraries.  It is inconvenient to use up the single
return value of a Java function with an error code.  (Java doesn't allow
function arguments to be passed by reference, so returning multiple
values tends to be more clumsy than in other languages.)

\subsubsection{Multiple Return Values.}
A few functions in the MPI interface return multiple values, even
after the error code is eliminated.  This is dealt with in mpiJava in
various ways.  Sometimes an MPI function initializes some elements in
an array and also returns a count of the number of elements modified.
In Java we typically return an array result, omitting the count.  The
count can be obtained subsequently from the {\tt length} member of the
array.  Sometimes an MPI function initializes an object conditionally
and returns a separate flag to say if the operation succeeded.  In
Java we typically return an object reference which is {\tt null} if
the operation fails.  Occasionally extra internal fields are added to an
existing MPI class to hold extra results---for example the {\tt
Status} class has an extra field initialized by functions like {\tt
Waitany} to hold the {\em index} value.  Rarely none of these methods
work and we resort to defining auxiliary classes to hold multiple
results from a particular function.

\subsubsection{Array Count Arguments.}
The mpiJava binding often omits array size arguments, because they can   
be picked up within the function by reading the {\tt length} member of
the array argument.  A major exception is for message buffers, where
an explicit count is always given.
%\begin{rationale}
In the mpiJava, message buffers have explicit {\tt offset} and
{\tt count} arguments whereas other kinds of array argument typically
do not.  Message buffers aside, typical array arguments to MPI
functions (eg, vectors of request structures) are small arrays.  If
subsections of these must be passed to an MPI function, the sections
can be copied to smaller arrays at little cost.  In contrast message
buffers are typically large and copying them is expensive, so it is
worthwhile to pass the extra arguments.   Also, if derived data types
are being used, the required value of the {\tt count} argument is
always different to the buffer length.
%\end{rationale}

\subsubsection{Concurrent Access to Arrays.}
In JNI-based wrapper implementations it may be
necessary to impose some non-interference rules for concurrent read
and write operations on arrays.  When an array is passed to an
MPI method such as a send or receive operation, the wrapper code will
probably extract a pointer to the contents of the array using a JNI
{\tt Get{\ldots}ArrayElements} routine.  If the garbage collector {\em does
not} support ``pinning'' (temporarily disabling run-time relocation of
data for specific arrays---see \cite{GORDON} for more
discussion), the pointer returned by this {\tt Get} function may be to
a temporary copy of the elements.  The copy will be written back to the
true Java array when a subsequent call to {\tt
Release{\ldots}ArrayElements} is made.  If two operations involving the
same array are active concurrently, this copy-back may result in
failure to save modifications made by one or more of the concurrent
calls.

Such an implementation may have to enforce a safety rule such as:  {\em
when several MPI send or receive (etc) operations are active
concurrently, if any one of those operations writes to a particular
array, none of the other operations must read or write {\em any
portion} of that array.}

If the garbage collector supports pinning, this problem does not arise.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Derived Datatype vs Object Serialization\label{DATATYPE}}

In MPI new \emph{derived types} of class \texttt{Datatype} can be
created using suitable library functions. The derived types allow one to
treat contiguous, strided, or indirectly indexed segments of program
arrays as individual message elements. The corresponding array
subsections can then be communicated in a single function call,
potentially exploiting any special hardware or software the platform
provides for exchanging scattered data between user space and the
communication system. 

Currently mpiJava provides all the derived datatype
constructors of standard MPI, with one limitation: it places significant
restrictions on its binding of \texttt{MPI\_TYPE\_STRUCT}. In C or
Fortran this function can be used to describe an entity combining fields
of different primitive (or derived) type. Because of the assumption that
buffers are one-dimensional arrays with elements of primitive type,
mpiJava imposes a restriction that all the types combined by
its \texttt{Datatype.Struct} member must have the same \emph{base type},
which must agree with the element type of the buffer array. Also
mpiJava does not provide an analogue of \texttt{MPI\_BOTTOM}
buffer address, or the \texttt{MPI\_ADDRESS} function for finding
offsets relative to this absolute member base. In C or Fortran these
functions allow buffers to include fields from separately declared
variables or arrays, but the mechanism does not sit well with the
pointer-free Java language model. 

Approaches based on the MPI derived datatype model do not seem to be the
best way to alleviate this restriction. A better option is probably to
exploit the run-time type information already provided in Java objects.
We have developed a version of mpiJava that adds one new predefined
datatype: \texttt{MPI.Object}. 
A message buffer can then be an array of any serializable Java objects.
The objects are serialized automatically in the wrapper of send
operations, and unserialized at their destination. 
We will discuss object serialization for Marshalling data in more
detail in Chapter \ref{chap:objseri}. 

The absence of true multi-dimensional arrays in Java limits another use
of derived data types. In MPI the \texttt{MPI\_TYPE\_VECTOR} function
creates a derived datatype representing a strided section of an array.
In C or Fortran this strided section can be identified with a section of
a multi-dimensional array. It could describe, say, an edge of the local
patch of a two-dimensional distributed array. In Java there is no
equivalence between a multi-dimensional array and a contiguous patch of
memory, or a one-dimensional array. The programmer may choose to
linearize all multi-dimensional arrays in the algorithm, representing
them as one-dimensional arrays with suitable index expressions. In this
case derived datatypes can be used to send and receive sections of the
array. Alternatively the programmer may use Java arrays of arrays to
represent multi-dimensional arrays. This simplifies the index arithmetic
in the program. Sections of the array are then explicitly copied to
one-dimensional buffers for communication. The latter option seems to be
more popular with programmers.

Although, for reasons of conformance of with MPI standards, we expect to
continue supporting derived datatypes in mpiJava, their value in the
Java domain is less clear-cut than in C or Fortran. Allowing
serializable objects as buffer elements is probably a more powerful
facility. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{mpiJava Implementations}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
To ``port'' mpiJava, it is
necessary to have a native MPI library, a version of the Java
Development Toolkit (JDK) and a C compiler. mpiJava consists of
two main parts: the MPI Java classes and the C stubs that binds the MPI
Java classes to the underlying native MPI implementation.  We create
these C stubs using JNI---the means by which Java can call and pass
parameters to and from a native API, see the next section for
details. Figure \ref{Layers} provides a 
simple schematic view of the software layers involved.

\begin{figure}[t]
\centerline{\epsfig{figure=Figs/Layers.eps,width=2in}}
\caption{Software Layers\label{Layers}}
\end{figure}

To port mpiJava onto a new platform, generally two steps are needed.

\begin{itemize}

\item Create a native library out of the compiled JNI C stubs.

\item Compile MPI Java in class libraries - ensuring that the correctly named
stub library is loaded by the Java
\texttt{System.loadLibrary("mpijava")} call in the main source file
\texttt{MPI.java}. 

\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\subsection{Java Native Interface\label{JNI}}
%The Java native interface is a great way to gain and merge the power
%of C or C++ programming into Java. 
%....
%The only programs user created are \texttt{.java} and \texttt{.c}
%. the \texttt{.h} and 
%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{UNIX Platforms\label{UNIX}}
The development and testing of mpiJava was undertaken on
various Sun and SGI UNIX platforms using MPICH.  
Interfacing Java to MPI is not always trivial. In earlier
implementations we often saw low-level conflicts between the Java
runtime and the interrupt mechanisms used in the MPI
implementations. The situation is improving as the JDK matures, in
particular version 1.2 allows the use of \emph{green} or \emph{native}
threads.
In JDK 1.1, Java threads are implemented on Solaris with a user-level
thread package known as green threads. In JDK 1.2, green threads are
the default, but the developer can optionally use Solaris native
threads for Java threads.  
With the green threads, a Java program creates and controls multiple
threads executing within a single CPU.  
On the other hand, native threads make it possible for Java programs
to take advantage of 
multiple CPUs on a single machine, are more scalable, and work better
with third-party native methods.  
%But in native mode, the system can schedule threads across multiple
%CPUs, significantly speeding processing. 
As a by-product this new \emph{native} thread feature has eliminated 
the interrupt problem that we encountered with earlier releases of the JDK. 
mpiJava is now stable on UNIX platforms using MPICH and JDK
1.2 .  



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{PC Platforms}
As Java is a platform-neutral language there is much interest in
porting mpiJava to PC-based systems, in particular Windows NT
and Linux. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Windows NT}
mpiJava is stable on NT platforms using WMPI and JDK 1.1
or later.  To test mpiJava under Windows NT we had the choice
of a number of MPI implementations to pick from \cite{BAKER}. We chose
WMPI \cite{WMPI} from the Instituto Supererior de Engenharia de
Coimbra, Portugal. WMPI is a full implementation of MPI for Microsoft
Win32 platforms. WMPI is based on MPICH and includes a p4
\cite{BUTLER} device standard.  WMPI can co-exist and interact with
MPICH/ch\_p4 in a cluster of mixed UNIX and Win32 platforms. WMPI is
still under development and is freely available.

To create a release of mpiJava for WMPI, the mpiJava JNI C interface
was compiled into a Win32 Dynamic Link Library (\texttt{mpiJava.dll}),
%Second, modify the name of the library loaded by the
%mpiJava interface (\texttt{MPI.java}) to that of the of the
%newly compiled library.
then the Java MPI interface was compiled into class libraries, and  
finally a JNI interface to WMPI was created. 
%This was necessary as under WMPI a master process is first
%spawned. Its purpose is to first read in a job configuration file and
%use the information within it to set up and run the actual MPI
%processes. An idiosyncrasy of WMPI is that all MPI processes must have
%a file name with the extension \emph{.EXE}. This led to the need to
%produce a JNI to WMPI so that the JVM was loaded and the {\em main}
%method of mpiJava Java class started.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Linux}

With previous version of JDK1.1.7 for Linux from Blackdown
\cite{BLACKDOWN}, we had problems similar to those encountered during
our early attempts to create the interface on Solaris, mentioned in
section \ref{UNIX}.  Sun releases the JDK for Solaris and NT
platforms first. On other platforms, such as Linux, it is necessary
for developers to port the JDK. 

mpiJava is now stable on Linux platforms using MPICH and
Blackdown's JDK1.2pre-v2 and IBM's JDK1.1.6 or later. 
It is also tested sucessfully with Sun's JDK1.2.2 for Linux.
The IBM alpha-version 1.1.6 of JDK \cite{IBMJAVALINUX} for
Linux was released in 1999. 
IBM JDK1.1.6 uses \emph{native} threads as the underlying
implementation. Blackdown and Sun JDK1.2 supports both \emph{green} and
\emph{native} threads.

%All Sun, Blackdown and IBM JDK require Linux kernel version 2.2 and glibc
%version 2.1 or greater.
The most recent release of the Blackdown JDK for
Linux is 1.2.2-RC4 (Release Candidate 4) that requires at least a
glibc-2.1.2 based system with a recent 2.2.xx kernel.

%LinuxThreads is the Linux POSIX threads implementation package built
%into glibc2 which is the GNU C runtime library version 2. In version
%2.0 of glibc, LinuxThreads did not have all the functionality required
%by the software. Version 2.1 of glibc has this required functionality,
%but some of this functionality is also dependent on the 2.2 version of
%the Linux kernel.

We have tested mpiJava with Mandrake 6.0 \cite{MANDRAKE},
Redhat 6.2 \cite{REDHAT} Linux.
These Linux packages incorporate both glibc version 2.1.x and Linux
kernel version 2.2.x.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Functionality Tests}

An integral part of the development of this project was to produce or
translate a number of basic MPI test codes to mpiJava. An
obvious starting point was the C test suite originally developed by
IBM . This suite had been modified to comply fully with the MPI
standard and to be compatible with the MPICH. The suite consists of
fifty-seven C programs that test the following MPI calls and data
types: collective operations, communicators, data types, environmental
inquiries, groups, point to point and virtual topologies. These codes
were all translated to mpiJava.

Under WMPI and Solaris-MPICH these codes were run either as multiple
processes on a single machine---``shared memory mode''---or as multiple
processes running on separate machines---``distributed memory mode''.
Under WMPI and Solaris-MPICH all the codes ran in both modes without
alterations. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Simple Communications Performance Measurement}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

At this early stage of our project we have decided to restrict
performance measurements to those that will give some indication of
the basic inter-processor communications performance. The actual
computational performance of each process is felt to be dependent on
the local JVM and associated technologies used by specific vendors to
increase the performance of Java. A real application performance will
be presented in Section  ~\ref{ISING_PERORMANCE}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{PingPong Communications Performance Tests }

In this program increasing sized messages are sent back and forth
between processes---this is commonly called {\em PingPong}. This benchmark
is based on standard blocking {\tt MPI\_Send} and {\tt MPI\_Recv}. PingPong
provides information about latency of {\tt MPI\_Send} and {\tt MPI\_Recv} and
uni-directional bandwidth. To ensure that anomalies in message timings
are minimized the PingPong is repeated many times for each message
size. The codes used for these tests were those developed by Baker and
Grassl \cite{PINGPONG}.  The three existing codes (MPI-C, MPI-Fortran
and Winsock-C) were used for comparison and we implemented an
mpiJava version for our purposes.

%The main problem encountered running the PingPong code was that under
%WMPI on Win32 \texttt{MPI\_Wtime()} had been implemented with a
%millisecond resolution. It was necessary to adapt each of the codes to
%use an alternative timer with microsecond ($\mu$s) resolution. 
The performance tests shown in the next section were run on two
similar systems:
\begin{itemize}
\item Two dual processor (P6 200 MHz) NT 4 workstations each with 128
MBytes of DRAM.

\item Two dual processor (UltraSparc 200 MHz) Solaris workstations with
256 MBytes of DRAM.
\end{itemize}
Both systems were connected via 10BaseT Ethernet and the tests were
carried out when there was little network activity and on quiet
machines.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Message Startup Latencies\label{sec:startup}}

\begin{table} [b]
  \begin{center}
    \leavevmode
    \small
    \begin{tabular}[c]{|l|l|l|l|l|l|}

    \hline
                    &
    \textbf{Wsock} &
    \textbf{WMPI-C} &
    \textbf{WMPI-J} &
    \textbf{MPICH-C} &
    \textbf{MPICH-J} \\
%    \textbf{Linux-C} &
%    \textbf{Linux-J} \\
    \hline

    \textbf{SM }&
    144.8 $\mu$s &
    67.2  $\mu$s &
    161.4 $\mu$s & 
    148.7 $\mu$s &
    374.6 $\mu$s \\
%    - $\mu$s &
%    - $\mu$s \\

    \hline

    \textbf{DM} &
    244.9 $\mu$s &
    623.9 $\mu$s &
    689.7 $\mu$s &
    679.1 $\mu$s &
    961.2 $\mu$s \\
%    - $\mu$s &
%    - $\mu$s \\

    \hline

    \end{tabular}
    \normalsize
    \caption{Time for 1 Byte Messages\label{latency}}
  \end{center}
\end{table}

In Table \ref{latency} we show the transmission time in microseconds
($\mu$s) to send a 1 byte message in each of the environments tested. In
shared memory mode (SM) the mpiJava wrapper adds an extra 94$\mu$s
(140\%) and 226$\mu$s (152\%) compared to WMPI and MPICH C
respectively. In distributed memory mode (DM) the mpiJava wrapper adds
an extra 66$\mu$s (11\%) and 282$\mu$s (42\%) compared to WMPI and
MPICH C respectively. The ``Wsock'' figures are those for a WinSock
implementation of PingPong benchmark using TCP.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Results in Shared Memory Mode (Figure \ref{fig5})}

The mpiJava curve mirrors that of C with an almost constant
offset up to 8K, thereafter the curves converge meeting at 256K. Under
MPICH, the curves for C and mpiJava mirror each other in a
similar fashion to those under WMPI, again there is a constant offset
and convergence at around 256K.

\begin{figure}[bt]
\centerline{\epsfig{figure=Figs/pingpong_sm.eps,width=140mm}}
\caption{PingPong Results in Shared Memory (SM) mode\label{fig5}}
\end{figure}

Under WMPI the peak bandwidth of C is around 65 MBytes/s and
mpiJava is 54 MBytes/s. The peaks occur at around 64K. Under
MPICH the bandwidth is flattening out, but still increasing for C and
mpiJava, at the 1M. The actual rate measured at this point is
about 50 MBytes/s.

Clearly the WMPI C code perform best of those tested. The performance
of mpiJava in SM under WMPI is good---it exhibits a fairly constant
overhead of 95$\mu$s up to 2K, thereafter it converges with the C
curve. The performance the C code under MPICH is slightly surprising
as the NT and Solaris platforms used for these tests had similar
specifications. It is assumed that the performance reflects the usage
of MPICH rather than a native version of MPI for Solaris. Even so, the
MPICH results for mpiJava show that it exhibits reasonable
performance.

\subsection{Results in Distributed Memory Mode (Figure \ref{fig6})}

\begin{figure}[bt]
\centerline{\epsfig{figure=Figs/pingpong_dm.eps,width=140mm}}
\caption{PingPong Results in Distributed Memory (DM) mode\label{fig6}}
\end{figure}

In distributed memory mode the differences between the MPI codes is
not as pronounced as seen in shared memory mode. Under WMPI the C and
mpiJava codes display very similar performance characteristics
throughout the range tested. Under MPICH, there is distinct
performance difference between C and mpiJava, However the difference
is much smaller than in SM and the curves converge at the 4K. All
curves peak at about 1 MByte/s, which is about 90\% of the maximum
attainable on 10 Mbps Ethernet link.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Discussion}

We have discussed the design and development of mpiJava---a
pure Java interface to MPI. 
Our performance tests have shown that mpiJava
should fulfill the needs of MPI programmers not only in terms of
functionality but also in terms of good performance when compared to
similar C MPI programs.  

In both shared memory and distributed memory modes mpiJava adds a
fairly constant overhead compared to normal native MPI. In an
environment like WMPI, which has been optimized for Windows NT, the
actual overheads of using mpiJava are relatively small at around
100ms. Under MPICH the situation is not quite so good, here the use of
mpiJava introduces an extra overheads of between 250--300$\mu$s.
It should be noted that these results compare codes running directly
under the operating system with those running in the JVM. For example,
according to a single 200 MHz PentiumPro will achieve in excess of 62
Mflop/s on a Fortran version of LinPack. A test of the Java LinPack
code gave a peak performance of 22 Mflop/s for the same processor
running the JVM. The difference in performance will account for much
of the additional overhead that mpiJava imposes on C MPI codes. The
quality and performance of JVM on each platform will have the greatest
effect on the usefulness of mpiJava for scientific computation.

We also suggested required environment to
make mpiJava run on different platforms including Linux.
On these platforms, several parallel applications including Potts
model simulation using Java AWT for GUI have been implemented. 
These applications will be covered in Chapter ~\ref{chap:applications} 
with their performance evaluation. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\section{Overall Summary}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%We have discussed the design and development of mpiJava - a
%pure Java interface to MPI. We have also highlighted the benefits of a
%fully object-oriented Java API compared to those currently
%available. Our performance tests have shown that mpiJava
%should fulfill the needs of MPI programmers not only in terms of
%functionality but also in terms of good performance when compared to
%similar C MPI programs.  We also suggested required environment to
%make mpiJava run on different platforms including Linux.
%We have implemented several parallel applications including Potts
%model simulation using Java AWT for GUI. 
%%Unfortunately, at
%%the time of submission of this paper we have been unable to test
%%mpiJava under Linux, but we believe that the problems we
%%have encountered we be overcome soon and we will be able to present
%%our finding during the Workshop in April 1999. 
%Overall, we feel that we have implemented a well designed, functional
%and efficient Java interface to MPI.
%

