Figure AC-1: Serialization with slim type encoding
DRAFT: This document is not yet finalized. Please do not quote.
By Dennis Gannon and George Thiruvathukal
The primary concern of the Java Grande Forum (hereafter JGF) is to ensure that the Java language, libraries and virtual machine can become the implementation vehicle of choice for future scientific and engineering applications. The first step in meeting this goal is to implement the complex and numerics proposals described in the report of the Numerics Working Group. Accomplishing this task provides the essential language semantics needed to write high quality scientific software. However, more will be required of the Java class libraries and runtime environment if we wish to capitalize on these language changes. The Java Grande Forum Applications & Concurrency Working Group (hereafter ACG) focuses on these issues.
It is possible that many of the needed improvements will be driven by commercial sector efforts to build server side enterprise applications. Indeed, the requirements of technical computing overlap with those of large enterprise applications in many ways. For example, both technical and enterprise computing applications can be very large and they will stress the memory management of the VM. The demand for very high throughput on network and I/O services is similar for both. Many of the features of the Enterprise Bean model will be of great importance to technical computing.
But there are also areas where technical computing is significantly different from Enterprise applications. For example, fine grained concurrency performance is substantially more critical in technical computing where a single computation may require 10,000 threads that synchronize in frequent, regular patterns. These computations would need to run on desktops as well as very large, shared memory multiprocessors. In technical applications, the same data may be accessed again and again, while in enterprise computing there is a great emphasis on transactions involving different data each time. Consequently, memory locality optimization may be more important for Grande applications than it is elsewhere in the Java world. Some technical applications will require the ability to link together multiple VMs concurrently executing on a dedicated cluster of processors which communicate through special high performance switches. On such a system, specialized, ultra low latency versions of the RMI protocol would be necessary. (In such an environment, an interface to shared memory, via RMI or the VM, would also be desirable.)
It is important to observe that there are problems which can be described as technical computing today which will become part of the enterprise applications of the future. For example, image analysis and computer vision are closely tied to application of data mining. The processing and control of data from arrays of sensors has important applications in manufacturing and medicine. The large scale simulation of non-linear mathematical systems is already finding its way into financial and marketing models. It is also the case that many technical computing applications do impact our day-to-day lives, such as aircraft simulation (the recent design of the Boeing 777) and weather forecasting. At least in the case of aircraft design, the industry has a valuation in the billions of dollars, which means it is far from being merely niche area being of limited interest.
This document is part of a larger report being written by the Java Grande
Forum members. There are two major sections of this
report: Numerics and Concurrency/Applications. For all practical purposes,
each of these documents is self-contained and thus can be read separately.
By Michael Philippsen and George Thiruvathukal
Sequential VM performance is of utmost importance to develop Grande applications. Since there are many groups working on this issue, the ACG simply provides some additional kernel benchmarks illustrating performance aspects in areas that are particularly important for Grande applications.
In addition to sequential VM performance, Grande applications require high performance for parallel and distributed computing. Although some more research is needed on other paradigms that might be better suited for parallelism in Java, this report will focus on RMI (Java's remote method invocation mechanism), since there is wide-spread agreement on both the general usefulness and the deficiencies of RMI.
In general, RMI provides the capability of allowing objects running in different JVMs to collaborate. Current RMI is specifically designed for peer-to-peer client/server-applications that communicate over the commodity Internet or over Ethernet. For high performance scientific applications, however, some of RMI's design goals and implementation decisions are inappropriate and cause serious performance limitations. This is especially troublesome on platforms targeted by the Java Grande community, i.e., closely connected environments, e.g. clusters of workstations and DMPs that are based on user-space interconnects like Myrinet, ParaStation, or SP2. On these platforms, a remote method invocation may not take longer than a few 10 microseconds.
The choice of a client/server model may prove too limited for many applications
in scientific computing, which usually take advantage of collective communication
as found in Message Passing Interface (MPI); however, the goal of this
document is not to propose such sweeping changes to RMI. We rather choose
to focus on how to make RMI, a client/server design, suitable for Grande
applications with no changes to the core model itself. It is our hope that
a better RMI design and implementation will stimulate community activities
to support better communication models that are well-suited to solving
community problems.
In an ideal solution, the exact byte representation of an object would be sent over the network and turned back into an object at the recipient's side without any unnecessary buffering and copying.
The ACG understands that the JDK's object serialization is used for several purposes, e.g. for long term storage of persistent objects and for dynamic class loading on remote hosts via http-servers. It is obvious that some of these special purpose uses require properties that are either costly to compute at runtime (latency) or that are verbose in their wire representation (bandwidth).
However, since some of these features are not used in Grande applications, there is room for improvement. The following subsections identify particular aspects of the current implementation of serialization that result in bad performance. The problems are described, and some solutions are suggested. Where possible, some benchmark results demonstrate the quantitative effects of the proposed solution.
Experiment: Experiments at Amsterdam [4] indicate that easily up to 30% of the run time of a remote method invocation are spent in the serialization, most of which can be avoided by compile time serialization.
When RMI uses the serialization for marshaling of method parameters, a new serialization connection is opened for every single method invocation. (More specifically, the reset method is called on the serialization stream.) Hence, type information is marshaled over and over again, thus consuming both latency and bandwidth. The current implementation cannot keep the connection open, because the serialization would otherwise refrain from re-sending arguments with modified instance variables. (Note, that the whole structure of objects reachable from argument objects is serialized; one of the objects deeply burried in that graph might have changed.)
Approach: For Grande applications it can be assumed that all JVMs that collaborate on a parallel application use the same file system, i.e., can load all classes from a common CLASSPATH. Furthermore, it is safe to assume that the life time of an object is within the boundaries of the overall runtime of the application. Hence, there is no need to completely encode and decode the type information in the byte stream and to transmit that information over the network.
Experiment: At Karlsruhe University, a slim type encoding has
been implemented prototypically [1]. It has improved
the performance of serialization significantly by avoiding the latency
of complicated type encoding and decoding mechanisms. Moreover, some bandwidth
can be saved due to the slimmer format. Figure AC-1 shows the runtime of
standard serialization in the first/blue row and the runtime of the improved
serialization with slim type encoding in the second/red row. The effect
is much more prominent on the side of the reader (right two bars, 2) than
on the side of the writer (left two bars, 1).
Figure AC-1: Serialization with slim type encoding
Solution: The ACG sees two options to avoid costly encoding of type information.
The conversion of these primitive data types into their equivalent byte representation is (on most machines) a matter of a type cast. However, in the current implementation, the type cast is implemented in the JNI and hence requires various time consuming operations for check pointing and state recovery upon JNI entry and JNI exit. Moreover, the serialization of float arrays (and double arrays) currently invokes the above mentioned JNI routine for every single array element.
Approach:
Experiment: A prototype at Karlsruhe University (see Figure AC-2,
[1])
stresses the effect of this approach.
Figure AC-2: Serialization of float arrays
Solution: It is absolutely essential that
In contrast to zero-copy protocols that are used (or attempted to be used) in other messaging protocols, where data is copied directly from user data structures to the network interface board, at least two complete copy operations (one at the sender side and one at the recipient side) are needed in every pure Java implementation. (Unfortunately, current implementations do much more copying that this.) Although the ACG regrets it, it seems very unlikely that future JVM implementations will closely interact with the communication mechanisms to allow for zero-copy protocols. This seems to be one of the price tags caused by Java's portability that can only be avoided by compile time serialization, see [4].
Approach: Obviously, there should be as few copy operations as possible. Those that remain should be performed as fast as possible.
Solution: Better performance can be achieved in two ways
Since the ACG does not believe that the communication buffer will ever be made public, the following proposal seems to be more promising.
ClassInfo getClassInfo(Field[] fields);
The object of type ClassInfo then provides two routines that do the copying to/from the communication buffer.
int toByteArray(Object obj, int objectoffset,
byte[] buffer, int bufferoffset);
int fromByteArray(Object obj, int objectoffset,
byte[] buffer, int bufferoffset);
The first routine copies the bytes that represent all the instance variables into the communication buffer (ideally the network interface board), starting at the given buffer offset. The first objectoffset bytes are left out. The routine returns the number of bytes that have actually been copied. Hence, if the communication buffer is too small to hold all bytes, the routine must be called again, with appropriately modified offsets.
Figure AC-3: Serialization of several instance variables
The ACG strongly recommends to allow for more efficient handling of the communication buffer, since performance of serialization can be increased by a factor of more than 10.
Experiment: In the current implementation, it is much faster to produce the byte array representation of an object than it is to recreate the object. The severity of this problem, that is probably caused by the overhead of object creation in Java, can be seen in Figures AC-1 to AC-3 above.
Approach: The ACG cannot provide any specific suggestions here.
Experiment: Experiments at Amsterdam [4] indicate that easily up to 30% of the run time of a remote method invocation are spent in the RMI implementation. Other 30% are spent in the network. By compiling Java code to native, a latency of a few 10 microseconds for a remote method invocation can be achieved on Myrinet. About the same performance is needed for jitted pure RMI code on custom interconnection hardware.
Approach: Whereas RMI's approach seems to be useful in case of unstable wide area networks, for closely connected JVMs this approach is far too pessimistic. Socket connections should be left open for the duration of the user program. At least, it might make more sense to keep a working set and use a least recently used algorithm to kick connections out of the working set.
Solution: To achieve this, the user should have an option to switch RMI into that mode. Again, this requires a command line option for the rmiregistry and another method in the RMISocketFactory class.
Approach: Instead of creating new objects and threads almost on every RMI activity, the internal layers of RMI should reuse sockets. Moreover, worker threads from a thread pool should repeatedly service incoming requests. The underlying asumption is that the network is quite stable and communication failure will cause failure of the whole application.
Problem: Operating systems typically allow one thread to monitor several socket connections by means of a select statement. There is currently no way for a Java thread to do the same. Instead, one Java thread is needed for every single open socket connection. This is not only a significant overhead for the JVM's thread sceduler but it prevents efficient implementations of socket based custom transports layers for RMI.
Solution: The JPACWG strongly recommends that JDK's sockets should provide a select statement. The RMI transport implementation should then make use of it.
Problem: It is almost impossible, to use specialized, high performance network protocols through RMI. Although SCI, ATM AAL5, Myrinet, ParaStation, Active Messages are all used in technical applications, there is no straightforward way for Java applications to make use of them.
The transport cannot be replaced individually, since the implementation does not properly isolate the transport layer. More specifically, Unicast source code has lots of type casts that explicity use TCP sockets and their methods. Hence, to get rid of the TCP transport, a completely new type of server needs to be implemented by hand as well.
Based on the current implementation and documentation of RMI, this work is comparable to re-inventing most of RMI's functionality. Although deeply burried in the internal layers of the RMI implementation, the transport seems to be plugged in very flexibly, there is no way that flexibility can be exploited without proper documentation.
There should be a way to make use of non-socket protocols in RMI. The following aspects need to be considered:
Since these protocols offer socket functionality, it seems to be easy to wrap them in a Java socket class that is then plugged into the RMI socket factory. Unfortunately, that approach does not work properly.
The problem is that SocketImpl returns file descriptors that are later on used by the JVM's thread scheduler. It issues calls on read/write/select methods which are only useful on the level of the operating system. For example, a select might be executed that waits for the arrival of a communication packet in the kernel although that packet has already arrived at the user level.
Hack around. The only reasonable way to go seems to be to load a dynamic library that replaces the standard operating system calls with those that can handle the high end communication hardware. This replacement is done unnoticed by the JVM. Such an approach has been successfully tested at the University of Karlsruhe on ParaStation hardware, it required an undesirable source code modification of the rmiregistry implementation (to load the dynamic library before main).
There are people who are willing to work on that code to improve it's runtime efficiency. Because of the missing source code (and the fact that de-compilers do not recreate the comments) the missing source code these people are prevented from doing work the ACG is interested in.
Solution: The ACG suggests that Sun will include the source code of current RMI implementations either in the standard JDK distribution or will create a process so that interested research groups can get access to it before a release gets final.
Solution: There may be no easy fix for this. Compatibility should probably be defined in terms of the version of Java being used as it is now; however, two classes with the same class and instance variables should be compatible, provided they are compiled with the same Java compiler.
Experiment: None known. Related to this problem is section 4.3 for which an experimental prototype has been constructed.
The notion of loading classes from the network is already known to be useful in the case of applets, where the code lives on the server in a code directory (often called the code "base") and can be downloaded to a web browser and loaded, subject to being verified by the class verifier. This is well known not to be without problems, because different browsers may not support the same Java (see Class Compatibility above); however, in principle it makes the life of a client easy.
In the case where a browser is not being used, it is still useful to consider the notion of a network class loader. In scientific and technical computing involving a large number of nodes (as found in massively parallel machines and clusters), the use of a network class loader can facilitate the deployment of code on a large number of nodes where there is not a shared file system (e.g., NFS or AFS). As it currently stands, when file systems are not shared, the programmer is forced to copy the class files to all nodes in a network where file systems are not shared. This can be quite cumbersome, although systems such as Unix provide remote commands to make the task easier. For Windows users, there is presently inadequate support for remote copying of files and remote execution, meaning that Windows file sharing must be used (still leaving one without adequate remote execution facilities). When heterogeneous systems are involved, the problem of deployment is even more complex, where everyone involved needs to know system administration.
Experiment: Thiruvathukal, Thomas, and Korczynski of Loyola
University and jhpc.org have shown how to extend the class loader for
a version of RMI called RRMI [3], where the client can start very thin and
most, if not all, classes are loaded from the network. The Globus
project has shown the importance of having facilities for remote
execution and job control.
By Martin Westhead
The purpose of developing a suite of benchmark tests is to provide ways of measuring and comparing alternative Java execution environments. In constructing this benchmark the aim is to provide a series of tests which challenge the execution environments in ways which are important to Grande applications. The benchmark has been divided into three sections:
EPCC in the University of Edinburgh is coordinating this effort, at time of writing, and are in the process of collecting existing benchmark tests to form a coherent suite. The aim of this collation is to have a package that can be downloaded and run as a whole, with consistent output format and a consistent definition of terms. This suite can then be added to over time as more and better tests are developed. The work being done at Edinburgh is ongoing and details can be found at:
The following sections elaborate on the proposed structure for the benchmark including examples of the tests that could be included. In most cases these tests are either available or have been promised.
It is anticipated that up to three different sizes of dataset should be available for section 2 (and maybe even two for section 3).
The user also should be able to run:
It seems unlikely that users would wish to run the entire benchmark suite in one go, but this facility could be added for completeness.
For timed benchmarks, the benchmarks will report a raw time (in seconds) and a performance measure in X/second, where the operation X is specific to the given benchmark (e.g. flops, method calls, iterations, etc.). Non-timing benchmarks will typically report a single figure (e.g. maximum size of object), but units should be consistent (all memory sizes in bytes, for example).
It is possible that such a system can be built on top of some of the existing and emerging meta-computing infrastructures. Many of these provide the tool kit and components to build on, and a few have partial solutions to the problems listed above.
Related projects are:
Such an interface should deal with the following issues: