\documentstyle[art11]{article}
\protect\input{psfig}
\textwidth 6.5in
\textheight 9.0in
\topmargin -.5in
\oddsidemargin -5pt
\evensidemargin -5pt
\newcommand{\comment}[1]{}
\def\sign{\mathop{\rm sign}\nolimits}
\newlength{\myindent}
\setlength{\myindent}{-0.3in}
\def\descriptionlabel#1{\hspace\labelsep #1}
\newenvironment{itemizer}{\begin{itemize}\setlength{\parsep}{0cm}\setlength{\itemsep}{-.3em}}{\end{itemize}}
\newenvironment{enumerator}{\begin{enumerate}\setlength{\itemsep}{0cm}}{\end{enumerate}}
\newenvironment{descriptor}{\begin{description}\setlength{\parsep}{0cm}\setlength{\itemsep}{-.2em}}{\end{description}}

\title{Java RMI Performance and Object Model Interoperability: Experiments
with Java/HPC++}

\author{Fabian Breg, Shridhar Diwan, Juan Villacis, \\ 
Jayashree Balasubramanian, Esra Akman, Dennis Gannon\\
Department of Computer Science \\
Indiana University
}

\date{}

\begin{document}
\maketitle


\begin{abstract}
Java RMI provides an elegant and powerful model for invoking
member functions on objects that exist in remote address spaces.  
%[JB: having Java objects talking to scientific applns. in C++/Fortran 
% seemed odd, hence ..]
Unfortunately, it is a Java-to-Java communication model, and in many of
the scientific applications we would like Java objects to interact with
modules written in C++ or Fortran.  This paper explores the design of RMI and
extracts a subset of the RMI object model that is compatible with C++ and
HPC++ remote object semantics.  This RMI subset has been implemented
to run over the Nexus runtime system and is being used as part of
the LSA distributed linear system analyzer project.
\end{abstract}

\section{Introduction}

Many high performance computing applications in the future will be designed
and built as distributed systems based on an object component architecture.
While some of the components in these systems will be built as Java clients
and servers, many others will be built using Fortran and C++.  In one
such system, the Linear System Analyzer (LSA) \cite{lsa}, about 22\% of the 
source is Java,  15\% is C++ and 63 \% is Fortran.  Java is used to program
the front end of the system which serves as the graphical environment
for composing computation components.  The components are Fortran modules
that run on sequential servers or, with the help of MPI, on parallel systems.
These modules are encapsulated with HPC++ \cite{hpc++lib}, which is a 
library based extension to C++ to support multi-threaded computation and 
remote method calls on distributed objects.  In the construction of the LSA 
we were presented with the following problem:  what is the best way for our 
Java application to communicate with remote objects written with HPC++?  
There were three obvious choices:  
\begin{enumerator}
\item Use a low level socket model or simple RPC mechanisms built on that.
\item Use the emerging Java CORBA\cite{corba} standard.
\item Find a way to allow the Java programmer to use a subset of the standard
Java Remote Method Invocation (RMI) mechanism that can interoperate with the
corresponding HPC++ remote object model.
% [JB: another possibility that must be discussed and renounced is JNI, 
% but we can leave that for later.. ]
\end{enumerator}
Our first implementation followed the first solution, but that approach proved
to be awkward.  While we have had extensive experience and success with CORBA,
many in our programming team found the Java RMI model to be richer and more
appealing.  Consequently, we undertook the task of implementing an RMI-to-C++
communication interface that would be consistent with the HPC++ remote
object model %[JB:that was ]
being used for communication between the C++/Fortran
components.  This paper describes what we have learned from this experiment.

There are two primary contributions of this work.
\begin{enumerate}
\item We explore the problem of finding the maximal subset of the Java RMI
object model that can be mapped into the remote object semantics of a C++
based language.  Our goal is to find those properties that are both powerful
and useful %, but also allow implementations that are efficient enough 
for high performance, scientific applications.

\item We illustrate, by means of a set of simple experiments, that Java RMI
exhibits very uneven performance.  While it is extremely fast in some
cases, it is subject to non-linear anomalies in others.  We demonstrate
that our Java-to-HPC++ prototype is able to avoid these performance problems.
\end{enumerate}

\section{A Java - HPC++ RMI System}
%[JB: Nice opener, but has to go for want of brevity]
%The Java RMI invocation system \cite{RMI} has become a de facto 
%standard for communication between different Java virtual machines (JVM).
Java RMI \cite{RMI} provides a mechanism for making method invocations on Java 
objects residing in different virtual machines.  Java RMI is specifically
designed to operate in the Java environment and so it seamlessly
integrates a distributed object model into the Java language.
%Because of this seamless integration, Java RMI is able to support most
%of the Java language's object semantics.[JB: An obvious conclusion]

While other RMI systems can be adapted to handle method invocations on remote
Java objects, these systems fall short of seamless integration with the Java
system due to their interoperability requirement with other languages.  For
example, CORBA presumes a heterogeneous, multi-language environment and thus
must have a language neutral object model.  In contrast, the Java 
language's RMI system assumes the homogeneous environment of the Java 
virtual machine, and the
system can therefore follow the Java object model whenever possible.
For example, RMI has the ability to send linked structures as arguments
in a remote invocation, and the ability to send an argument whose type is a
subclass of the formal parameter type.  These features are either missing in
CORBA or are difficult to implement.
%RMI assumes the dynamic class loading functionality provided by the JVM. 
%Thus it is able to provide certain features that are either lacking in 
%CORBA or are difficult to implement.  e.g. RMI allows invocation of 
%method calls that send an actual parameter whose type is a subclass of
%the formal parameter [ reference to section]. RMI is also able to send 
%linked data structures as parameters, and parameters that are references 
%to a single object [reference to section]. CORBA provides a facility by
%which a method invocation can send any type of actual parameter for a
%formal parameter declared of type {\tt Any}. But this facility lacks the
%ease of programming afforded by the above mentioned {\em subclass parameter}
%facility provided by RMI. CORBA doesn't provide any mechanism to send
%linked data structures or for sending parameters that refer to the same
%object. Thus, the object model supported by Java RMI is much richer than the
%CORBA object model.  

As our goal is to implement remote method invocations between Java and C++,
from a Java programmer's point of view, we want to preserve as much of the
Java language's object semantics as possible.  By implementing our own 
version of Java RMI to HPC++ communication system, we are able to support
an object model that, in terms of richness, lies between the object 
semantics of the Java language and that of CORBA.

As a first step towards implementing a Java RMI to HPC++ communication system,
we implemented a Java to Java RMI system on top of Nexus Java \cite{nj}, 
a Java interface to the Nexus communication library \cite{nexus}. 
This system is called Nexus RMI.  Next we  extended the communication 
mechanism underlying HPC++ so that it became compatible with the Nexus RMI
communication mechanism.  This established the communication link between 
Nexus RMI and HPC++.

The rest of the section is organized as follows.  We start with a brief
overview of the Java RMI system, followed by a description of Nexus RMI.
Next we give a brief description of HPC++ and related concepts that are 
relevant to communication with Java.  Finally, we describe our 
implementation of the Java RMI to HPC++ communication system, and 
the issues of language semantics involved in the implementation.

\subsection{Java RMI}

Java RMI is designed to simplify the communication between two objects in
different virtual machines by allowing an object to invoke the methods of an
object in another virtual machine, in the same way as methods on local objects
are invoked.  To invoke methods on a remote object, a remote reference to that
object has to be obtained.  Since the object resides in a different virtual
machine and hence in a different name space, a registry is used to manage 
remote references.  RMI servers can register their objects at the registry 
after which clients can obtain a reference to these remote objects. 
A more detailed description of the RMI architecture can be found in [10].

%[JB: neat description of Java RMI but again we will assume everyone
%knows what RMI is all about] First a server
%creates a remote object and registers it at the registry, which is shown by
%arrows (1) in the figure. The client can obtain references to objects
%stored in the registry as shown by arrows (2). When the client invokes a 
%method on the remote object, the method is actually invoked 
%(shown by arrow (3)) on a stub object, located in the same JVM as the 
%client's JVM. The stub object makes a message containing the name of 
%the method together with its parameters (called {\em marshaling}), 
%and sends this message (shown by arrow (4)) to the associated skeleton
%object residing in the server's JVM. The skeleton object extracts the 
%method name and parameters from the message (called {\em unmarshaling}),
%and invokes the appropriate method on the remote object with which it is 
%associated (shown by arrow (5)). The remote object
%executes the method and passes the return value back to the skeleton 
%(arrow (6)). The skeleton in turn marshals the return value in a message
%and sends this message to the stub object (arrow (7)).  The stub unmarshals
%the return value from the message and returns this value to the client 
%program (Arrow (8)). 

% -------------------------------------------------------------- 
%\begin{figure}
%\centerline{\psfig{figure=rmi.eps,height=3.0in}}
%\caption{Java RMI architecture.}
%\end{figure}
% -------------------------------------------------------------- 

%The methods of a remote object that can be invoked remotely must be 
%specified in an interface that extends the RMI defined {\tt Remote} 
%interface. Every method in the interface must be declared to throw 
%the {\tt RemoteException} in order to account for errors during 
%remote method invocation. A number of subclasses of 
%{\tt RemoteException} are defined that represent the various errors 
%that can occur.

%A remote object is typically declared to implement at least one remote 
%interface and also to extend the {\tt RemoteServer} class or one of its 
%subclasses. After creating a remote object it needs to be exported which 
%makes it available for remote access. Once created and exported, a remote
%object must be registered with a registry by invoking either the {\tt bind}
%or {\tt rebind} method of the {\tt Naming} class,
%providing them with the remote object and an appropriate name. 

%The remote interface of a remote object is used as the type of the remote
%reference on the client side. A remote reference can be obtained by invoking
%the {\tt lookup} method of the Naming class, providing the name of the 
%desired remote object. When invoking a method through the remote interface,
%the {\tt RemoteException} must be caught to handle the errors during the 
%remote method invocation. Exceptions that occur during the actual execution
%of the remote method at the server are thrown to the client encapsulated in a
%{\tt ServerRuntimeException} object. Other exceptions occurring at the server
%are encapsulated in a {\tt ServerException} object.

%After writing and compiling the remote interface and the remote object
%implementation, the RMI stub/skeleton compiler, {\tt rmic}, can be used
%to generate a stub and a skeleton pair for accessing the remote object.
%The stub and the skeleton objects and thus the entire communication is 
%completely shielded from the programmer.

\subsection{Nexus RMI}

Nexus RMI is our implementation of Java RMI on top of Nexus Java, which 
is a Java interface to the Nexus communication library \cite{NexusRMI}.
The Nexus communication library provides dynamic resource management, 
multi-threading and multiple methods for communication, allowing it to
operate in a heterogeneous environment.  Nexus Java currently implements 
a subset of Nexus in Java.

The interface to Nexus Java is organized around six basic abstractions. 
A {\em node} represents a physical processing resource.  On a
node, multiple {\em contexts} can be running, which can be considered to 
be equivalent to a JVM.  In each context, multiple {\em threads} can be present.
Communication is performed over a communication link which is created by 
binding a communication {\em startpoint} to a communication {\em endpoint}.
To invoke methods on objects associated with an endpoint, a 
{\em remote service request} (RSR) can be issued on the startpoint connected 
to the endpoint.  When the RSR is issued on a startpoint, a message 
containing a {\em message handler identifier} and a data buffer are sent over the
communication link to the endpoint, after which the method specified by the
handler identifier is invoked providing it with the buffer as a parameter. 
Nexus Java is compatible with the C/C++ language interface to the Nexus 
library.  Thus, Nexus Java can be used to invoke RSRs on contexts that are 
either other JVMs or C/C++ contexts.

Nexus RMI is implemented on top of the basic RSR mechanism provided by
Nexus Java.  When a server binds a remote object into the registry, a startpoint
referring to the remote object is also provided to the registry.  When a
client does a lookup for the remote object, the registry provides it with the
startpoint for the remote object and this startpoint is stored inside the stub
object.  When the client invokes a method on the stub, the stub uses this
startpoint and the Nexus Java RSR mechanism to forward the request over to
the skeleton object in the server JVM.

The complete RMI system consists of the RMI communication protocol (which
includes support for object serialization) and the stub and skeleton
compiler.  The classes in our implementation can be divided into classes
that implement the client-server communication, classes that implement the
registry, and the exception classes.  The exception classes in our 
implementation are copied directly from the RMI specification.

%Implementing the RMI protocol using Nexus Java involved implementing
%the classes from the RMI specification and providing support for object
%serialization. Object serialization deals with sending the internal state of
%objects as a message to another JVM and reconstructing that object from the
%received message. Only objects that extend the {\tt Serializable} interface can
%be serialized. As the state of the object includes its private and protected
%members, and as other objects cannot access this part of the state, public read
%and write methods need to be added to the serializable objects to perform the
%actual serialization.  [JB: This is an implementation detail of Nexus RMI we 
%probably could omit for the short version]

In Nexus RMI, we have implemented the original RMI classes, thus implementing
the RMI communication protocol.  Nexus RMI also has support for object
serialization of all object types except for exception objects.  It
provides a complete stub compiler implementation, which generates stubs and
skeletons for remote objects.  It also adds the serialization methods to
serializable classes.  Although Nexus RMI has support for remote exceptions, 
the implementation is not complete yet.  Also, distributed garbage collection
is not implemented yet.

\subsection{HPC++}

HPC++ is a C++ library and language extension framework that is being 
developed by the HPC++ consortium as a standard model for portable parallel
C++ programming. %The design has been partitioned into two levels. level 1 
%consists of a specification for a set of class libraries and tools that do
%not require any extensions to the C++ language. Level 2 provides the basic
%language extensions and runtime library needed to implement the full Level
%1 HPC++ specification.[%JB Again details about what HPC++ is expendable]
The current HPC++ framework describes a C++ library along with compiler
directives which support parallel C++ programming. 

The standard architecture model supported by HPC++ is a system composed of a set
of interconnected nodes.  Each node is a shared-memory multiprocessor (SMP) and
may have several contexts, or virtual address spaces.

The central problem associated with multi-context computation is the
communication and synchronization of events between two contexts.  HPC++
is based on the CC++ \cite{cc++} global pointer concept and the library
implements this with a $GlobalPtr<T>$ template as is done in the MPC++ 
Template Library \cite{MPC++}.

A global pointer generalizes the C pointer type to support 
pointers to objects that exist in other address spaces.  It is closely 
linked to the idea of a global reference which is an object that acts as
proxy for a remote object.  A global pointer can be used to access the
remote object that it points to, or it can also be used to make method
invocations on the remote object.  So, a global pointer is similar
to a stub object in Java RMI.

For a user defined class C with member function {\tt foo}, the standard 
way to invoke {\tt foo} through a pointer is as follows: 

{\small
\begin{verbatim} 
class C {
public:
  int foo(float, char);
};
C *p;                    
p->foo(3.14, 'x');
\end{verbatim}
}

It is a bit more work to make the member function call through a global
pointer. First, all members that will be called through global pointers 
need to be registered with the HPC++system as shown below:

{\small
\begin{verbatim} 
int result;
hpcxx_id_t C_foo_id = hpcxx_register(C::foo, id_value);
HPCxx_GlobalPtr<C> P;
hpcxx_invoke(P, result, C_foo_id, 3.14, 'x');
\end{verbatim}
}

%The overloaded {\tt hpcxx\_register} template function builds a table 
%for system information about registered functions, classes, and member
%functions, and returns its location as an ID. The arguments to the 
%registration is the function to be registered and the index key for 
%table look-up. Because this ID is an index into the table, it is 
%essential that each context register the members with the same value 
%for the table index.  

To invoke the member function {\tt foo}, the special function template 
{\tt hpcxx\_invoke} that calls {\tt C::foo(3.14, 'x')} in the context 
which contains the object pointed to by {\tt P}.  The calling process 
waits until the function returns.

HPC++ is implemented on top of the Nexus communication library, making it
compatible with the communication mechanism of Nexus RMI.  A global pointer
maintains a Nexus startpoint that points to the actual remote object.  When 
{\tt hpcxx\_invoke} is called on a global pointer, the function uses 
the startpoint inside the global pointer to invoke an RSR on the remote
object's context.

\subsection{ The Java RMI - HPC++ Communication system}

\subsubsection{ Basic Communication Infrastructure}

The communication endpoint in Java RMI is the stub object that acts as a
surrogate for the remote object in another JVM. If Java RMI is to communicate
with HPC++, the stub object must act as a surrogate for some remote object
located in an HPC++ context.
%As our implementation of Java RMI, i.e. Nexus RMI,
%is based on using Nexus startpoints (inside stub objects) to identify the 
%remote object, the straightforward approach is to let this startpoint point
%to an HPC++ object rather than a Java remote object. When the client invokes
%a method on the stub, as the stub's startpoint points to an HPC++ object, 
%the method call is forwarded to an HPC++ object.
%[JB: Nexus RMI and its communication model has already been explained.
% Hence the above rephrase]
By having the startpoint in the Nexus RMI stub object point to 
an HPC++ object rather than a Java remote object, method calls made on the
Java object are forwarded to the HPC++ object.

The first step in implementing this communication mechanism is to use a 
common registry between HPC++ and Nexus RMI. We used the Nexus RMI
registry as the common registry. Nexus RMI already has a {\tt Naming} 
class implemented that can be used to lookup and bind remote objects. 
So on the HPC++ side, we implemented a similar {\tt Naming} class for
registering HPC++ objects in the Nexus RMI registry. The {\tt bind} 
function of the {\tt Naming} class takes a global pointer to
an HPC++ object and binds it in the registry. The global pointer is 
bound with an appropriate user provided name. As a result of the 
{\tt bind} function call, the startpoint pointing to the HPC++ object
is stored in the registry. When a Java client invokes {\tt lookup} 
with the object's name, the registry provides the startpoint that 
points to the HPC++ object. This startpoint is kept in the client 
stub object thus letting the stub communicate with the HPC++ object.
%As the HPC++ server uses the Nexus RMI registry, and as the Nexus RMI
%client uses the same interface to get object references to Java and 
%HPC++ objects, the Nexus RMI client is unaware of the fact that it's
%communicating with an HPC++ server object. 
The {\tt Naming} class for
HPC++ also contains a {\tt lookup} method, so that HPC++ clients can 
lookup Java remote objects (or HPC++ objects for that matter) and 
invoke methods on them. 

\subsubsection{Issues in Java RMI to HPC++ communication}

Although being able to forward method requests from Java to HPC++ objects is a
good first step, the whole process involves deeper semantics issues because of
the differences in object models of Java and C++. This subsection discusses
the problems we faced in implementing the Java RMI to HPC++ communication
link.  It also describes some as yet unresolved problems, and our proposed
solutions for them.

\paragraph{Exception Handling.}

HPC++ has an exception model based on the CORBA exception model. 
There are two kinds of exceptions:
\begin{itemize}
\item system exceptions: these correspond to standard runtime errors
which may occur during the execution of a request.
\item user exceptions: user defined exceptions resulting from executing
user code in the implementation of a method.
\end{itemize}

The system exceptions are predefined for an HPC++ implementation and are
compiled into the HPC++ library. The user exceptions must be available to the
HPC++ system at compile time.
%and if the user wants to add a new kind of user
%exception, she will have to compile appropriate portions of the user code to
%accommodate the new exception. 

On the other hand, a Java function can throw exceptions that may be dynamically
loaded and hence not known at compile time. So when a Java remote object sends
back an exception to the Java client, the client may have to dynamically load
the exception class before throwing the exception to the client code. This is
implemented in Nexus RMI by requiring the remote object to marshal the fully
qualified name of the exception class along with the exception object while
sending it to the client.

These differences in the exception model lead to different issues
when a Java client invokes a method on an HPC++ object, and vice-versa:
\begin{itemize}
\item Java to HPC++ method call: As all HPC++ exceptions are available at
compile time, they can be mapped into appropriate Java RMI exceptions. 
The skeleton code on the HPC++ side maps the exception thrown by the 
HPC++ object method into an appropriate RMI exception before sending 
it back to the Java side. 
%The skeleton code marshals the fully qualified
%name of the exception class into the message buffer before marshaling 
%the exception object.

\item HPC++ to Java method call: When the HPC++ client receives an
exception from a Java object, it receives the fully qualified name of
the exception class first. The HPC++ system maps this name into an 
appropriate HPC++ exception and then throws this exception to the 
client code. We have implemented a mapping from standard RMI exceptions
to HPC++
exceptions. If the remote Java object throws an exception unknown to the HPC++
system, it just thrown an HPC++ exception called {\tt UnknownException}. If the
client were a Java client, it would have been able to load the new exception
class and throw the proper exception. This limitation will probably be 
faced by any system that allows communication between Java and C++. 
\end{itemize} 

\paragraph{ Sending an actual parameter that is a subclass of the formal
parameter in a remote method request.}


Consider the following piece of Java code:
{\small
\begin{verbatim}
public class D implements Serializable { ... }   // parameter class
public Class E extends D { ... }                 // parameter class

public interface CIface extends Remote {         // a remote interface
  int fun(D param1, D param2) throws RemoteException;
}

class C extends UnicastRemoteObject implements CIface { // implementation class
  public int fun(D param1, D param2) { ... } 
}
\end{verbatim}
}

\noindent Referring to the code above, the client can invoke a method as follows:

{\small
\begin{verbatim}
CIface cref;
cref = (CIface)Naming.lookup(...); // lookup for a remote object implementing CIface

D arg1 = new D;
E arg2 = new E;
int ret = cref.fun(arg1, arg2); // arg2 is of type E
\end{verbatim}
}

Here, an RMI request can send an argument whose type is a subclass of
the formal parameter class. 
%This poses no problem for communication between
%a Java client and a Java remote object, as the skeleton on the remote 
%object side can always load the argument class dynamically, instantiate
%it, and invoke the request on the remote object. All that the skeleton
%code needs is the fully qualified name of the actual parameter, which
%can provided by the client in the message buffer. Nexus RMI uses this
%strategy. 
However, this will not work if the remote object is an
HPC++ object. Since standard C++ cannot load classes dynamically (without 
depending on system specific dynamic loading functionality), HPC++ is 
limited to only the classes known to it at compile time.  This is a
major restriction when invoking methods from Java to C++.  Furthermore,
the Java side cannot send an argument that is a subclass of the parameter
class.  This can be enforced by defining the parameter class {\tt final}, so 
that the Java client cannot create subclasses of the parameter class.
%The current Java RMI - HPC++ communication system mandates this
%restriction. 

One way to implement this functionality for HPC++ is to use the dynamic loading
facility provided by the underlying operating system to load objects, classes
and functions at runtime.  For example, on Solaris, the calls {\tt dlopen} 
and {\tt dlsym} can be used to dynamically load and access shared objects.

%e.g. Solaris provides the call {\tt dlopen} 
%to load a {\em shared object} into the process address space, and the 
%call {\tt dlsym} to access a specific symbol, either function or data 
%symbol, from the shared object. This mechanism can be used to provide 
%the same kind of functionality as RMI provides.

Another way to implement a restricted form of the above functionality is to
require the user to enumerate all the possible types that may be used as actual
parameters in the place of a formal parameter. A stub compiler can then be used
to generate appropriate code that handles the case when the received
parameters' type is a subclass of the type of the formal parameter.

\paragraph{An argument in a method call that is a reference to another argument.}

Referring back to the Java code above, consider the following method
invocation:

{\small
\begin{verbatim}
D arg1 = new D;
D arg2 = arg1;
int ret1 = cref.fun(arg1, arg2);
\end{verbatim}
}

Here, both {\tt arg1} and {\tt arg2} 
refer to the same object. In order to preserve the semantic equivalence
of local requests and remote requests, the RMI system sends only one copy
of the argument to the remote object.  When the method {\tt fun} is
invoked on the remote object, {\tt arg1} and {\tt arg2} both are 
references to the same object.  It is difficult to maintain this
semantic equivalence if the remote object is an HPC++ object rather 
than a Java object.  If the method {\tt fun} on the HPC++ side has the
signature:

{\small
\begin{verbatim}
int fun(D arg1, D arg2);
\end{verbatim}
}

\noindent then {\tt arg1} and {\tt arg2} cannot be references to the same object.
However, if instead {\tt arg1}, {\tt arg2} are made references to {\tt D}, 
then we may be able to invoke the method on the remote C++ object with
{\tt arg1} and {\tt arg2} both pointing to the same object.  But 
implementing  Java references as C++ references may not always work 
due to the restrictive nature of C++ references.  For example,
if the parameter sent is an array with some of the array elements 
referring to the same object, this solution fails since C++ cannot create
an array of references.

%if there is another method {\tt sendarray} on the Java side:  
%
%{\small
%\begin{verbatim}
%// a remote interface
%public interface CIface extends Remote {
%  int fun(D param1, D param2) throws RemoteException;
%  void sendarray(D [] myarr) throws RemoteException;
%}
%// implementation class
%class C extends UnicastRemoteObject implements CIface {
%  public int fun(D param1, D param2) {
%  ...
%  }
%  void sendarray(D [] myarr) {
%  ...
%  }
%}
%\end{verbatim}
%}
%
%Then it is possible to invoke the method {\tt sendarray} from a Java 
%client such that two elements of the array {\tt myarr} point to the 
%same element. This is possible because {\tt myarr} is essentially an
%array of $references$ to {\tt D} type objects rather than an array of
%{\tt D} type objects. It is impossible to maintain semantics equivalence
%on the C++ side, if {\tt myarr} is implemented either as an array of {\tt D}
%type objects or as an array of references to {\tt D} type objects. 
%An array of {\tt D} type objects will not work for the reason explained
%above in case of function {\tt fun}. An array of references will not 
%work either, as you cannot create an array of references in C++. 
%So the only way left is to define an array of pointers to {\tt D} type objects. 
%So the method {\tt sendarray} will look like this on the C++ side:
%
%{\small
%\begin{verbatim}
%void sendarray(D *myarr[]);
%\end{verbatim}
%}

%This is probably mandatory, as on the Java side an array of $references$ to
%objects of type {\tt D} is essentially an array of $pointers$ to objects 
%of type {\tt D}.

So the only way this problem can be solved is by implementing Java 
references as C++ {\em pointers}.  This strategy works even for an 
array type parameter 
because on the Java side an array of $references$ to objects of type {\tt D} is
essentially an array of $pointers$ to objects of type {\tt D}.

Nexus RMI already has a mechanism to deal with this problem of object reference
aliasing.  On the sender side, on every remote method invocation, a table is
constructed out of objects as and when they are encountered during the
marshaling procedure. Whenever an object is to be marshaled into the buffer, 
a check is made to see whether the object is already in the table.  If it 
is not, it is marshaled into the buffer and also inserted into the table.  
If it is in the table, this means that the object was previously marshaled, 
and so only the object's index in the table is marshaled into the buffer. 
Since buffers in Nexus Java have a
first in - first out property for their contents, exactly the same table is
reconstructed at the receiving side.  While unmarshaling the buffer, if an 
object is found, it is inserted into the table and the receiving object 
reference is set to this unmarshaled object.  On the other hand, if an 
index is found, the receiving object reference is set to the object at the 
table entry at the index received.

Our current implementation of Java RMI - HPC++ communication system does not
address the problem of multiple arguments referring to the same object.  It
requires the Java side to invoke methods such that the arguments
do not reference the same object, if the receiving side is HPC++.  A possible
solution is to implement the above Nexus RMI scheme on the HPC++ side.

%We are in the process of implementing a general purpose mechanism to deal with
%the problem of reference aliasing. It will involve all object references on the
%Java side being represented as pointers on the HPC++ side and then implementing
%the above mentioned mechanism utilized by Nexus RMI.

\paragraph{Sending linked structures as arguments.}

In Java RMI, it is possible to send a linked list of objects just by
sending one element as an argument; the semantics dictates that the entire
linked list must be recreated on the receiving side.

Again, it is difficult to maintain this semantics if the receiving side 
is a C++ object.  For example, CORBA does not support this semantics mainly
because in CORBA, an object used as a parameter cannot contain a pointer to
another object. 

We currently do not have support for this feature in our implementation of Java
RMI - HPC++ communication system.  The way we can implement this mechanism on
the HPC++ side is by implementing a smarter unmarshaling algorithm
that can unmarshal objects (containing pointers) recursively.

%But these are exactly the kinds of facilities
%we intend to implement so that we will be able to support a richer object model
%than that provided by CORBA. 
%This will require a smarter unmarshaling algorithm on the HPC++ side. 
%While unmarshaling an object that contains a pointer, the unmarshaling algorithm
%not only has to unmarshal the object but also the object that the pointer points
%to. 

\paragraph{Member Classes.}
Both in Java and C++, a class can have another class as a member class. 
There is a slight difference in the semantics of member classes for the 
two languages.  In C++, a member function of the inner class cannot 
directly access the data members of the outer class; while in Java such
an access is permitted.  We feel that this difference in semantics will
not impose any restrictions on the object model that our communication
system supports. 

\pagebreak

\section{Experimental Analysis}

\subsection{Overview of Experiments}

We set up a simple $ping-pong$ test in which identical random data of
a given size was passed between a client and server program through a
remote method call.  Using the measured value of the round-trip delay we 
calculated throughput as,
%time difference immediately before and after client invocation of 
%the remote method.
\[
         TP = (totalBits)/(totalRoundTripTime)
\]

TP was measured for variable-size arrays of primitive and object types. 
The primitive data type was taken to be an 8-byte {\tt double}. The object 
data type had the form,

{\small
\begin{verbatim}
public class MyDouble {
  double value;
}
\end{verbatim}
}

\noindent The size of this structure was assumed to be the same as for a
{\tt double} (i.e., 8 bytes).

\subsection{Implementation Environment}

Our experiments tested two implementations of remote method 
invocation (Java RMI using a 1.1 based JVM and our Nexus RMI)
versus a purely HPC++ implementation.  Although we performed
experiments on several architectures, time and space considerations
allow us to present only three.  The experimental data presented here
% was obtained by executing client and server programs on two identical 
% IBM RS/6000 604-based PowerPC quad-processor machines running AIX4.2
% and connected via a 10-Mbit Ethernet LAN.  We used IBM's JIT-optimized 
% JDK1.1.2 compiler to run our Java programs, and compiled our C/C++ 
% programs using xlC with -O optimization.  Performance on Solaris and
% SGI systems exhibited similar behavior.
was obtained by measuring client/server throughput between two similarly
configured machines and for three types of network connections:  
10-Mbit ethernet, 155-Mbit ATM, and loopback.
In the ethernet case, we used two identical IBM RS/6000 117MHz 604-based 
powerpc quad-processor machines equipped with 256MB RAM and running AIX4.2 .  
We used IBM's JIT-optimized 
JDK1.1.2 compiler to run our Java programs, and compiled our C/C++ 
programs using {\tt xlC} with {\tt -O} optimization.  In the ATM case, 
we used two dual-processor ultrasparcs (one operating at 200MHz with 
256MB RAM, the other at 300MHz with 512MB RAM)
and each running Solaris2.6 .  We used Sun's JIT-optimized JDK1.1.3 compiler
to run our Java programs, and compiled our C/C++ programs using the
SunPro {\tt cc} compiler with {\tt -O} optimization.  In the loopback case, 
we used the 200MHz dual-processor ultrasparc with the same OS and compilation 
parameters as in the ATM case.

Due to the wide ranges in our data, we present graphs of throughput
versus array size in $loglog$ format for each experiment described below.

\subsection{Experiments}

In this section we describe each of the four experiments in some detail.
The first experiment involved using strictly Java to Java communication
via Java RMI.  The second experiment also involved Java to Java communication
but used Nexus RMI protocol instead.  The third experiment involved a
combination of Nexus RMI and HPC++ objects.  The fourth and final 
experiment measured a purely HPC++ communicating object system.

\paragraph{Java RMI.}

For the Java RMI experiment, we used Java's {\tt System.currentTimeMillis()}
to measure time.  Although the time resolution was in milliseconds,
our tests involved network latencies larger than a millisecond, and
so Java's timing method was sufficient for our purposes.

In addition to measuring overall round-trip time, we also inserted
timings within the stub and skeleton sources (as generated by Sun's
{\tt rmic} tool) to measure time spent marshaling and unmarshaling the
data.  Since our remote method simply returned its argument, we
expected the total time to be,

\begin{equation}Time = CMT + SUT + SMT + CUT  \end{equation}

where
\begin{center}
\begin{tabular}{lcl}
{\em Time} &=& total roundtrip time \\
{\em CMT} &=& client marshal time \\
{\em SUT} &=& server unmarshal time \\
{\em SMT} &=& server marshal time \\
{\em CUT} &=& client unmarshal time \\
\end{tabular}
\end{center}

To our surprise, we found that marshaling and unmarshaling times
overlapped, i.e., data was being {\em pipelined} rather than passed
serially.  This meant that total time in Java RMI was effectively,

\begin{equation}Time = max(CMT, SUT) + max(SMT, CUT)  \end{equation}

\paragraph{Nexus RMI.}

%Nexus RMI aims to mimic the semantics of the Java RMI remote method call
%o that Java RMI programs can interoperate with non-Java programs via
%he Nexus protocol.  [JB: Detailed explanation of NexusRMI already provided in
%section 2.1]
%Although there are restrictions as to the kinds
%of objects that can be passed between the two models, our narrowly-defined
%tests fell safely within those constraints. [ JB: we have already mentioned that
%NexusRMI is a subset of Java RMI]

In the Nexus RMI experiment, we use our own prototype RMI compiler
to generate stub and skeleton Java code that uses the Nexus Java from Argonne.
Again, we inserted timings within the generated stub/skeleton codes in order
to measure marshaling/unmarshaling times.  The timings we measured
followed the equation given in (1).  This was expected since Nexus
explicitly blocks until data is either ready to send or ready to read
from its internal buffers.

Our first experiments with the array of objects demonstrated
the complexity of Java object serialization.
The algorithm used to marshal and unmarshal the data structures in
our compiler had $quadratic$ complexity in the number of objects marshaled.
Consequently, performance fell very rapidly  in the case of an array
of objects.  After modifying the algorithm, the drop in performance was
eliminated.

\paragraph{Nexus RMI/HPC++.}

In this experiment, we used a Java client to invoke the remote method
(via Nexus RMI) on an HPC++ server following the method described in
the previous subsection.

\paragraph{HPC++.}

In this experiment, we had an HPC++ client invoke a remote method
(via Nexus) on an HPC++ server.  The timings were obtained using
Unix {\tt getrusage} system call.

%The results of these experiments are illustrated in Figure 2.

% -------------------------------------------------------------- 
%\begin{figure}
%\centerline{\psfig{figure=prim_obj_plot.ps,height=3.5in}}
%\caption{Performance in Mbits/second of TP for both the array of
%primitives and the array of objects for each of the four cases.}
%\end{figure}
% -------------------------------------------------------------- 

\subsection{Performance Analysis}

Using the data in the first graph of Figure 1, we compared the 
relative performance of each method on the array of primitives over
ethernet.  We see that Java RMI outperforms HPC++,
Nexus RMI/HPC++, and Nexus RMI for sufficiently large data arrays.
Since data is pipelined in Java RMI, but not in Nexus RMI, this result
is reasonable.  What is more surprising is that when the JIT is enabled,
Java RMI is able to outperform HPC++.

% -------------------------------------------------------------- 
\begin{figure}[h]
\centerline{ \psfig{figure=etherprm.eps,height=2.6in} 
\psfig{figure=etherobj.eps,height=2.6in}}
\caption{{\small Ethernet TP performance for arrays of primitive type {\tt double}, and object type {\tt MyDouble}}.}
\end{figure}
% -------------------------------------------------------------- 

\pagebreak

The second graph of Figure 1 illustrates the relative performance for all
methods on the array of objects over ethernet.  Here, Java RMI 
{\em under-performs} all the others.  One reason might be that 
marshaling/unmarshaling {\em objects} is a
very costly operation since RMI must maintain Java object reference
semantics across different JVMs (see Section 2.4.2).

%\begin{figure}
%\centerline{\psfig{figure=obj_plot.ps,height=3.5in}}
%\caption{Performance in Mbits/second of TP for an array of type MyDouble.}
%\end{figure}

In comparing the TPs of both graphs in Figure 1,
we see that the performance of Java RMI w/objects is several orders of
magnitude worse than Java RMI w/primitives.  Also, Java RMI w/objects
exhibits a strange downturn in performance as array sizes increase.

We performed a few more experiments to narrow down the possible causes
for the poor performance and anomalous behavior in Java RMI w/objects.

% -------------------------------------------------------------- 
\begin{figure}[h]
\centerline{\psfig{figure=serobj.eps,height=3.0in}}
\caption{Java RMI object array performance with different 
serialization methods.}
\end{figure}
% -------------------------------------------------------------- 

In the first experiment, we tested whether inefficient default 
serialization methods were to blame.  For example, since we did not 
specify {\em how} to serialize {\tt MyDouble},
the system {\tt defaultReadObject()} and {\tt defaultWriteObject()} methods
were used for object serialization.  
In this experiment, we varied the degree of control a user has
in the serialization process.  First, we allowed for some user control
by adding the {\tt readObject()} and {\tt writeObject()} private methods to
{\tt MyDouble}.  Next, we enforced maximum user control in the serialization 
process by adding the {\tt readExternal()} and {\tt writeExternal()} 
public methods to {\tt MyDouble}.  In both instances, the added methods 
had empty bodies so that we could see the minimum effect they might have 
on the TP.  The results of this experiment are shown in Figure 2.  
We discovered that although there was some improvement in the TP, the
increase was less than a factor of 2.  Furthermore, the {\em shape} of each
graph was the same as the original Java RMI w/objects.  

% -------------------------------------------------------------- 
\begin{figure}[h]
\centerline{ \psfig{figure=atmprimo.eps,height=2.6in} 
\psfig{figure=atmobj.eps,height=2.6in}}
\caption{{\small ATM TP performance for arrays of primitive type {\tt double}, and object type {\tt MyDouble}}.}
\end{figure}
% -------------------------------------------------------------- 

In the second experiment, we tested how the TP scaled with increasing 
bandwidth.  We first ran each object system over a high bandwidth ATM 
network.  The results of this experiment are shown in Figure 3.  
We found that although each object system took advantage of the increased 
bandwidth, Java RMI w/objects didn't improve much over the ethernet case.  
Next, we eliminated the effect the network might have on TP by executing
both client and server on the same machine.  In this case, the client/server
communication went through the machine's loopback interface.  The results 
of this experiment are shown in Figure 4.  We found that even when
communication costs are factored out, the overhead due to object 
serialization still limits the maximum TP that can be achieved.

% -------------------------------------------------------------- 
\begin{figure}[h]
\centerline{ \psfig{figure=lpbkprim.eps,height=2.6in} 
\psfig{figure=lpbkobj.eps,height=2.6in}}
\caption{{\small Loopback TP performance for arrays of primitive type {\tt double}, and object type {\tt MyDouble}}.}
\end{figure}
% -------------------------------------------------------------- 

From this we have concluded that an end user cannot significantly improve 
the performance of Java RMI w/objects, and that the performance problems 
in Java RMI are inherent to the serialization process itself.

It should be noted that in all cases involving Java, tests with array sizes 
over $10^5$ could not be consistently performed due to either memory 
limitations in the JVM (e.g., could not allocate sufficient heap size),
or the entire array would not fit in RAM (thus causing too much variability
in timing due to swapping).

\section{Conclusion}

In this paper we have illustrated how a subset of the Java RMI object model 
can be mapped onto the HPC++ remote object system. Global references in 
Java map to global pointers in HPC++lib and {\tt final} object classes map to
C++ classes with little ambiguity.  However, the semantic mapping between
these object systems has limitations.  For example, parts of Java object
serialization, exception handling and certain types of aliased references
cannot be duplicated in HPC++. The most serious problems relate to the 
inability of standard C++ to dynamically load classes or to use reflection.
In future work we plan to provide a more formal mapping by using an
extended CORBA IDL to define the interface and semantics of the RMI
subset more concretely. 

The performance experiments presented here illustrate several important 
points. First, we demonstrate that Java RMI is very good at managing modest
size arrays of primitive types on {\em commodity} networks such as Ethernet.  
Due to the pipelining mechanism implemented by RMI, Java was able to attain a
better TP than HPC++ which is designed as a substrate for high performance
scientific applications. This further suggests that communication libraries
and runtime systems (such as Nexus) designed for high performance applications
can use RMI's pipelining strategy to make a substantial gain in TP.  Although
problems were encountered with very large arrays, we do not consider this a
serious matter.  However, there are major concerns regarding the performance
of RMI with arrays of object and on high performance networks such as ATM.  
The first problem is that RMI shows a non-linear behavior while transferring
large arrays of objects.  Because of the more limited semantics of our RMI
subset, we were able to obtain superior performance with our implementation.  
On the other hand, we feel that there is nothing about the RMI semantics that
requires a non-linear algorithm for serialization and we can expect to see
improvements in future versions.  The other problem is that RMI is unable to
exploit the high bandwidth afforded by high performance networks.  As
demonstrated by our experiment, either for the primitive or for the object 
array case, RMI performs quite poorly as compared to HPC++.  Our experiments
also suggest that the bottleneck lies in the object serialization part.  
We feel that RMI is currently unsuitable for high performance applications 
due to its poor performance on high bandwidth networks.

%The performance experiments presented here illustrated several important 
%points. First, we demonstrate that Java RMI is very good at managing modest
%size arrays of primitive types.  Due to the pipelining mechanism implemented 
%by RMI, Java was able to attain a better TP than HPC++, which is designed as
%a substrate for high performance scientific applications.  This further 
%suggests that the communication libraries and runtime systems (such as Nexus)
%designed for high performance applications can use RMI's pipelining strategy 
%to make a substantial gain in TP. Though problems were encountered with very
%large arrays, we do not consider this a serious matter.  However, of greater 
%concern is the problem associated with the lack of linearity demonstrated 
%in moving large arrays of objects with Java RMI. Because of the more 
%limited semantics of our RMI subset, we were able to obtain superior 
%performance with our implementation.  On the other hand, we feel that 
%there is nothing about the RMI semantics that requires a non-linear 
%algorithm for serialization and we can expect to see improvements in 
%future versions. 

In the future, we plan to repeat these tests over a wider range of 
high bandwidth networks and we will do a deeper analysis of the actual 
protocols.  Our goal is to be able to allow high performance components 
of a distributed application to communicate with low latency and 
high bandwidth. It is our feeling that the performance of the current 
generation of Java and CORBA distributed object systems lags far behind
the requirements of scientific and engineering application.  
However, we are confident that Java based solutions will eventually
emerge as the solution.

\section*{Acknowledgments}

We would like to thank Aart J.C. Bik who is the author of the {\tt javar}
compiler \cite{javar} and various routines that read bytecode representations,
which we used as a basis for our Nexus RMI compiler

%\pagebreak

\bibliographystyle{plain}
\bibliography{java}  % Bib refs are in a file "java.bib"

\end{document}
