%src.hpj
%  |
%ExpMgr :
%ClassRef :    Table
%LocalVar :
%  |
%TypeChecking
%  |
%Translator
%  |
%src.java
%
\chapter{HPJava: A Java Environment for SPMD Programming
\label{chap:hpjava}} 

\pagestyle{myheadings}
\markright{CHAPTER\ref{chap:hpjava}.  HPJAVA}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The Java message-passing library {\em mpiJava} discussed in the
remaining of this dissertation is part of a larger environment called
{\em HPJava}. 

Early in the last decade, 
two groups formed for developing standards for parallel
programming. 
The first, the High Performance Fortran Forum was formed by many
leading industrial and academic groups in 1992.  They established a
language standard called High Performance Fortran (HPF) that allows
programmers to easily write data-parallel programs.
About a year later, the second group, the Message Passing Interface
Forum, with participation from over 40 organizations, collected ideas
from many previous message-passing systems and put them into a
standard called MPI.

HPF provide a simple programming model in which all
processors execute a single, logical thread of control that performs
high-level operations on distributed arrays.  
In contrast, MPI provide multiple, logical threads of control that
excute their own tasks in their own processors.
%over the mapping of a parallel program's tasks to processors.   
HPF is compiler-based and MPI is library-based.

MPI has been quite successful and widely used for programming parallel 
system. On the other hand, HPF has been less successful, and many 
companies have abandoned their HPF projects. 
One of the failings is that HPF is not easily compatible with SPMD
programming using MPI or many other similar libraries

Unlike HPF, our HPJava environment is based on an {\it HPspmd} model
that is designed to facilitate direct calls to established SPMD
libraries, including message-passing libraries, for parallel
programming.  To increase the flexibility of the system, and make it
more attractive to programmers accustomed to the direct
message-passing style, an interface to MPI is clearly essential.

The main emphasis is of this dissertation is mpiJava.
mpiJava itself does not depend on any special language extension. 
For completeness, however, this chapter will review the language model
of the full HPJava environment. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Motivation}

Data parallel programming languages have always held a special position
in the high-performance computing world.  The basic implementation
issues related to this paradigm are well understood.  However, the
choice of high-level programming environment, particularly for modern
MIMD architectures, remains uncertain. In 1993, the High
Performance Fortran Forum published the first standardized definition
of a language for data parallel programming
\cite{HPFStandard,HPFBOOK}.  In the intervening period considerable
progress has been made in HPF compiler technology, and the HPF language
definition has been extended and revised in response to demands of
compiler-writers and end-users \cite{HPF2Standard}.  Yet it seems to be
the case that most programmers developing parallel applications---or
environments for parallel application development---do {\em not} code
in HPF.  The slow uptake of HPF can be attributed in part to immaturity
in the current generation of compilers.  But it seems likely that many
programmers are actually more comfortable with the Single Program
Multiple Data (SPMD) programming style, perhaps because the effect of
executing an SPMD program is more controllable, and the process of
tuning for efficiency is more intuitive.

SPMD programming has been very successful.  There are
countless applications written in the most basic SPMD style, using
direct message-passing through MPI or similar
low-level packages.  Many higher-level parallel programming
environments and libraries assume the SPMD style as their basic model.
Examples include ScaLAPACK \cite{SCALAPACK}, PetSc \cite{PETSc1}, DAGH
\cite{HDDA_DAGH}, Kelp \cite{Block_structured}, the Global Array
Toolkit \cite{Global_Arrays} and NWChem \cite{NWChem1}.  While
there remains a prejudice that HPF is best suited for problems with
very regular data structures and regular data access patterns, SPMD
frameworks like DAGH and Kelp have been designed to deal directly with
irregularly distributed data, and other libraries like CHAOS/PARTI
\cite{CHAOS} and Global Arrays support unstructured access to
distributed arrays.  

These successes aside, the library-based SPMD
approach to data-parallel programming certainly lacks the uniformity
and elegance of HPF.  All the environments referred to above have some
idea of a distributed array, but they all describe those arrays
differently.  Compared with HPF, creating distributed arrays and
accessing their local and remote elements is clumsy and error-prone.
Because the arrays are managed entirely in libraries, the compiler
offers little support and no safety net of compile-time checking.

This chapter discusses a class of programming languages
that borrow certain ideas, various run-time technologies, and some
compilation techniques from HPF, but relinquish some of its basic
tenets.  In particular they forgo the principles that the programmer
should write in a language with (logically) a single global thread of
control, and that the compiler should determine automatically which
processor executes individual computations in a program, then
automatically insert communications if an individual
computation involves accesses is to non-local array elements.

If these assumptions are removed from the HPF model, does anything
useful remain?  What will be retained is an explicitly
MIMD (SPMD) programming model complemented by syntax for representing
distributed arrays, and syntax for expressing that certain computations
are localized to certain processors, including syntax for a distributed
form of the parallel loop.  The claim is that these features
are adequate to make calls to various data-parallel libraries, including
application-oriented libraries and high-level libraries for
communication, about as convenient as, say, making a call to an array
transformational intrinsic function in Fortran 90.  Besides their
advantages as a framework for library usage, the resulting programming
languages can conveniently express various practical data-parallel
algorithms.  The resulting framework may also have better prospects for
dealing effectively with {\em irregular} problems than is the case for
HPF.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{HPspmd Language Extensions}

HPspmd system aims to provide a flexible hybrid of the data parallel
and low-level SPMD paradigms.  To this end HPF-like distributed arrays
appear as language primitives.  But a design decision is made that all
access to {\em non-local} array elements should go through library
functions---either calls to a collective communication library, or
simply {\tt get} and {\tt put} functions for access to remote blocks
of a distributed array.  Clearly this decision puts an extra onus on
the programmer; but making communication explicit encourages the
programmer to write algorithms that exploit locality, and simplifies
the task of the compiler writer.

For the newcomer to HPF, one of the advantages of that language lies
in the fact that the effect of a particular operation is logically
identical to its effect in the corresponding sequential program.
Assuming programmers understand conventional Fortran, it is very easy
for them to understand the behavior of a program at the level of what
values are held in program variables, and the final results of
procedures and programs.  Unfortunately, the ease of understanding
this ``value semantics'' of a program is counterbalanced by the
difficulty in knowing exactly how the compiler translated the program.
Understanding the {\em performance} of an HPF program may require that
the programmer have rather detailed knowledge of how arrays are
distributed over processor memories, and what strategy the compiler
adopts for distributing computations.

The language model we discuss has a special relationship to the HPF
model, but the HPF-style semantic equivalence between the data-parallel
program and a sequential program is abandoned in favor of a simple
equivalence between the data-parallel program and an MIMD (SPMD) program.
Because understanding an SPMD program is presumably more difficult than
understanding a sequential program, our language may be slightly
harder to learn and use than HPF.  But understanding performance of
programs should be much easier.

The distributed arrays of an HPspmd language should be kept strictly
separate from ordinary arrays.  They are a different kind of object,
not type-compatible with ordinary arrays.  A property of the languages
we describe is that if a section of program text {\em looks like}
program text from the unenhanced base language (Fortran 90 or Java, for
example), it is translated exactly as for the base language---as local
sequential code.  Only statements involving the extended syntax are
treated specially.  This makes preprocessor-based implementation of the
new languages straightforward, allows sequential library code to be
called directly, and gives programmers good control over the generated
code---they can be confident no unexpected overhead have been
introduced into code that looked like ordinary Fortran, for example.

Our HPspmd system adopts a distributed array model
semantically equivalent to to the HPF data model in terms of how
elements are stored, the options for distribution and alignment, and
facilities for describing {\em regular sections} of arrays.
Distributed arrays may be subscripted with global subscripts, as in
HPF.  But an array element reference must not imply access to a
value held on a different processor. To simplify the task of the
programmer, who must be sure accessed elements are held locally, the
languages can add {\em distributed control} constructs.
These play a role something like the {\tt ON HOME} directives of HPF
2.0 and earlier data parallel languages \cite{Kali}.  One special
control construct---a distributed parallel loop---facilitates traversal
of locally held elements from a group of aligned arrays.

The HPJava is a Java instantiation language of this HPspmd language
model. A brief review of the syntax extensions is given in section
\ref{sec:HPJava}.   
In \cite{Bindings} we have outlined possible syntax
extensions to Fortran to provide similar semantics to HPJava. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Integration of High-Level and Low-Level Libraries\label{libraries}}

Libraries are at the heart of our HPspmd model.  From one point of
view, the language extensions are simply a framework for invoking
libraries that operate on distributed arrays.  
Hence an essential component of the ongoing work is definition of a series of
bindings from HPspmd languages to established SPMD libraries and
environments.  Because the language model is explicitly SPMD, such
bindings are a more straightforward proposition than in HPF, where
one typically has to pass some extrinsic interface barrier before
invoking SPMD-style functions.

We can group the existing SPMD libraries for data parallel programming
into three categories.
In the first category we have libraries like ScaLAPACK \cite{SCALAPACK}
and PetSc \cite{PETSc1} where the primary focus is similar to
conventional numerical libraries---providing implementations of
standard matrix algorithms (say) but operating on elements in regularly
distributed arrays.  We assume that designing HPspmd interfaces to
this kind of package will be relatively straightforward.
ScaLAPACK for example, provides linear algebra routines for
distributed-memory computers.  These routines operate on distributed
arrays---specifically, distributed matrices.  The distribution formats
supported are restricted to two-dimensional block-cyclic distribution
for dense matrices and one-dimensional block distribution for
narrow-band matrices.  Since both these distribution formats are
supported by HPspmd, using ScaLAPACK routines from the HPspmd framework should
present no fundamental difficulties.

In a second category we place libraries conceived primarily as
underlying support for general parallel programs with regular
distributed arrays.  They emphasize high-level communication primitives
for particular styles of programming, rather than specific numerical
algorithms.  These libraries include compiler runtime libraries like
Multiblock Parti \cite{structured} and Adlib \cite{PCRC_based}, and
application-level libraries like the Global Array toolkit
\cite{Global_Arrays}.  Adlib is a runtime library that was
designed to support HPF translation.  It provides communication
primitives similar to Multiblock PARTI, plus the Fortran 90
transformational intrinsics for arithmetic on distributed arrays.  The
Global Array (GA) toolkit, developed at Pacific Northwest National Lab,
provides an efficient and portable ``shared-memory'' programming
interface for distributed-memory computers.  Each process in a MIMD
parallel program can asynchronously access logical blocks of
distributed arrays, without need for explicit cooperation by other
processes (``one-sided communication'').  Besides providing a more
tractable interface for creation of multidimensional distributed
arrays, our syntax extensions should provide a more convenient
interface to primitives like {\tt ga\_get}, which copies a patch of
a global array to a local array.

Regular problems (such as the linear algebra examples in section
\ref{sec:HPJava}) are an important subset of parallel applications, but
of course they are far from exclusive.  Many important problems involve
data structures too irregular to represent purely through HPF-style
distributed arrays.  Our third category of libraries therefore includes
libraries designed to support irregular problems.  These include CHAOS
\cite{CHAOS} and DAGH \cite{HDDA_DAGH}.  We anticipate that
irregular problems will still benefit from regular data-parallel
language extensions---at some level they usually resort to
representations involving regular arrays.  But lower level SPMD
programming, facilitated by specialized class libraries, is likely to
take a more important role.  For an HPspmd binding of the CHAOS/PARTI library,
for example, the simplest assumption is that the preprocessing phases
yield new arrays.  Indirection arrays may well be left as HPspmd
distributed arrays; data arrays may be reduced to ordinary Java arrays
holding local elements.  Parallel loops of an executor phase can then
be expressed using {\em overall} constructs.  More advanced schemes may
incorporate irregular maps into generalized array descriptors
\cite{HPF2Standard,FortranD,ViennaFortran} and require extensions to
the baseline HPspmd language model.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{The HPJava Language\label{sec:HPJava}}

HPJava \cite{HPJava} is an instance of the HPsmpd language
model.  HPJava extends its base language, Java, by adding some
predefined classes and some additional syntax for dealing with
distributed arrays.

As explained in the previous section, the underlying distributed array
model is equivalent to the HPF array model.  Array mapping
is described in terms of a slightly different set of basic
concepts.  {\em Process group} objects generalize the processor
arrangements of HPF.  {\em Distributed range} objects are used
instead HPF templates.  A distributed range is comparable with a single
dimension of an HPF template.  These substitutions are a change
of parameterization only.  Groups and ranges fit better with our
distributed control constructs.

\begin{figure*}[tbp]
\small
\Line 
\begin{verbatim}
  Procs2 p = new Procs2(P, P) ;
  on(p) {
    Range x = new BlockRange(M, p.dim(0)) ;
    Range y = new BlockRange(N, p.dim(1)) ;

    float [[,]] a = new float [[x, y]], b = new float [[x, y]],
                c = new float [[x, y]] ;

    ... initialize values in `a', `b'

    overall(i = x for :)
      overall(j = y for :)
        c [i, j] = a [i, j] + b [i, j] ;
  }
\end{verbatim}
\Line 
\normalsize
\caption{A parallel matrix addition.\label{fig:addition}}
\end{figure*}

Figure \ref{fig:addition} is a simple example of an HPJava program.  It
illustrates creation of distributed arrays, and access to their
elements.  The class {\tt Procs2} is a standard library class derived
from the special base class {\tt Group}.  It represents a
two-dimensional grid of processes.  Similarly the distributed range class
{\tt BlockRange} is a library class derived from the special class {\tt
Range}; it denotes a range of subscripts distributed with
BLOCK distribution format over a specific process dimension.  Process
dimensions associated with a grid are returned by the {\tt dim()}
inquiry.
The {\tt on(p)} construct is a new control construct specifying that
the enclosed actions are performed only by processes in group {\tt p}.

The variables {\tt a}, {\tt b} and {\tt c} are all distributed array
objects.  The type signature of an $r$-dimensional distributed array
involves double brackets surrounding $r$ comma-separated slots.  The
constructors specify that these all have ranges {\tt x} and {\tt y}---they
are all {\tt M} by {\tt N} arrays, block-distributed over {\tt p}.

A second new control construct, {\em overall}, implements a
distributed parallel loop.  The constructs here iterates over all
locations (selected by the degenerate interval ``\verb$ : $'') of
ranges \verb$x$ and \verb$y$.  The symbols {\tt i} and {\tt j} scoped
by these constructs are called {\em distributed indices}.  In HPF, a
distributed array element is referenced using integer subscripts, like
an ordinary array.  In HPJava, with a couple of exceptions noted
below, the subscripts in element references must be distributed
indices, and these must be bound by an overall construct to a location
in the range associated with the array dimension.  This rather drastic
restriction is a principal means of ensuring that referenced array
elements are held locally.


\begin{figure}[t]

\small
\Line 
\begin{verbatim}
  Procs2 p = new Procs2(P, P) ;
  on(p) {
    Range x = new ExtBlockRange(N, p.dim(0), 1, 1) ;
    Range y = new ExtBlockRange(N, p.dim(1), 1, 1) ;

    float [[,]] u = new float [[x, y]] ;

    ... some code to initialise `u'

    for(int iter = 0 ; iter < NITER ; iter++) {

      Adlib.writeHalo(u) ;

      overall(i = x for 1 : N - 2)
        overall(j = y for 1 + (i` + iter) % 2 : N - 2 : 2)
          u [i, j] = 0.25 * (u [i - 1, j] + u [i + 1, j] +
                             u [i, j - 1] + u [i, j + 1]) ;
    }
  }
\end{verbatim}
\Line 
\normalsize
\caption{\label{fig:redBlack}Red-black iteration.}
\end{figure}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\afterpage{\clearpage}
\begin{figure}[t]
\small
\Line 
\begin{verbatim}
  Procs1 p = new Procs1(P) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;

    float [[,*]] a = new float [[x, N]], c = new float [[x, N]] ;
    float [[*,]] b = new float [[N, x]], tmp = new float [[N, x]] ;

    ... initialize `a', `b'

    for(int s = 0 ; s < N ; s++) {

      overall(i = x for :) {

        float sum = 0 ;
        for(int j = 0 ; j < N ; j++)
          sum += a [i, j] * b [j, i] ;

        c [i, (i` + s) % N] = sum ;
      }

      // cyclically shift `b' (by amount 1 in x dim)...

      Adlib.cshift(tmp, b, 1, 1) ;
      HPspmd.copy(b, tmp) ;
    }
  }
\end{verbatim}
\Line 
%\noindent \mbox{}\hrulefill\mbox{}
\normalsize
\caption{A pipelined matrix multiplication program.\label{fig:pipedMatmul}}
\end{figure}

The general policy is relaxed slightly to simplify coding of stencil
updates.  A subscript can be a {\em shifted location}.  Usually
this is only legal if the subscripted array is declared with suitable {\em
ghost regions} \cite{Ghost}.  Figure \ref{fig:redBlack} illustrates the
use of the library class {\tt ExtBlockRange} to create arrays with
ghost extensions (in this case, extensions of width 1 on either side of
the locally held ``physical'' segment).  The communication library
routine {\tt Adlib.writeHalo} updates the ghost region.  This example
also illustrates application of a postfix back-quote operator to a bound
location.  The expression {\tt i`} (read ``i-primed'') yields the integer
global loop index.

Distributed arrays can be defined with some sequential dimensions.  The
sequential attribute of an array dimension is flagged by an asterisk in
the type signature.  As illustrated in Figure \ref{fig:pipedMatmul},
element reference subscripts in sequential dimensions can be ordinary
integer expressions.


The short examples here have already covered much of the special syntax
of HPJava.  Other significant extensions allow Fortran-90-like {\em
sections} of distributed arrays.  This, in turn, forces us to define
certain {\em subranges} and {\em subgroups}.  Arrays constructed
directly using subgroups and subranges can reproduce all the alignment
options of HPF.  In any case, the language itself is relatively
simple.  Complexities associated with varied and irregular patterns of
communication are dealt with in libraries.  These can implement many
richer operations than the {\tt writeHalo} and {\tt cshift} functions
of the examples.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{MPI Programming in HPJava}


%\afterpage{\clearpage}
%\afterpage{}
\begin{figure}[t]
\small
\Line 
\begin{verbatim}
  Procs1 p = new Procs1(P) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;

    float [[]] f = new float [[x]], a = new float [[x]],
               b = new float [[x]] ;

    // initialize
    overall(i = x for :) {
      f[i] = 0 ;
      b[i] = a[i] ;
    }

    for(int s = 0 ; s < N ; s++) {

      overall(i = x for :) 
        f[i] += force(a[i], b[i]) ;

      // cyclically shift `b' (by amount 1 in x dim)...

      Adlib.cshift(tmp, b, 1, 0) ;
      HPspmd.copy(b, tmp) ;
    
    }
  }
\end{verbatim}
\Line 
\normalsize
\caption{HPJava data parallel version of the N-body.
        \label{fig:data-Nbody}}
\end{figure}

To increase the flexibility of the system, and make it more attractive
to programmers accustomed to the direct message-passing style, an
interface to MPI is clearly desirable.  In HPF, with its
global-thread-of-control model, a proper interface to the underlying
message-passing platform is only practical through the {\it extrinsic
procedure} mechanism.  Unlike HPF, the HPJava environment is based on
the HPspmd model which is designed to facilitate direct access to
SPMD library interfaces.
%HPJava provides special constructs to facilitate both data-parallel and
%task-parallel programming.  
Different processors can either simultaneously work on data with the
globally subscripted arrays, or independently execute complex
procedures on their own local data. The conversion between these phase
is relatively seamless.

We use a simple example extracted from an N-body algorithm to illustrate
the usage of the HPJava binding for MPI.
Figure ~\ref{fig:data-Nbody} is a simplified pure data parallel
version of force computation in a N-body program. 
There are three distributed arrays in the program, {\tt a}, {\tt b} and
{\tt f}. Distributed array {\tt a} has fixed copy of the particle
positions and {\tt b} has circulating copy of the positions

Now suppose we wanted to combine features of this HPJava
version with an MPI version, while partially keeping the convenient
data-parallel array syntax.
Figure ~\ref{fig:data-MPI} shows HPJava/MPI version of N-body.
Process group arrangements and initialization for distributed arrays
are handled by HPJava, while the shift-operation is done
by calling the mpiJava {\tt MPI.Sendrecv\_replace} point-to-point
communication routine between processors.  The local variables {\tt
a\_block}, {\tt b\_block} and {\tt f\_block} in the program are not
distributed arrays, but they are assigned different values by
distributed arrays {\tt a}, {\tt b} and {\tt f} according 
to their position in the process grid.   
A inquiry function {\tt dat()} returns a sequential Java array
containing the locally held elements of the distributed array.  The
array size of local variables for each processor is $B = N/P$, where
$N$ is total particle size and $P$ is number of processors.  These
local arrays run serially on each node, but concurrently across all the
nodes.

Comparing this HPJava/MPI version with the data-parallel version,
notice that both programs send the same amount of total data, but the
HPJava/MPI version does $P$ shifts of whole blocks of size $B$ for
sending $N$ data, while the pure data-parallel version uses a {\tt
cshift} call that does one circular shift for every step for sending
$N$ data.  This adds extra communication start-up overheads. In
addition, HPJava version requires more copying operations ($N$ times)
than the mpiJava version does ($P$ times), where typically $P \ll N$.
%In these examples, pure data-parallel program does N shifts by 1
%place.  On the other hand, HPJava/MPI program does P shifts of whole
%blocks.  This reduces communication between nodes.
%and then achieves good performance improvement.

This example leaves non-trivial issues unresolved---in general what is
the mapping from distributed-array elements to local-data-segment
elements? What is the mapping between HPJava process groups and MPI
groups? The complete specification of HPJava will address these
issues; eventually a better integrated message-passing API may be
desirable.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\afterpage{\clearpage}
\begin{figure}[t]
\small
\Line 
\begin{verbatim}
  Procs1 p = new Procs1(P) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;

    float [[]] f = new float [[x]], a = new float [[x]],
               b = new float [[x]] ;

    // initialize
    overall(i = x for :) {
      f[i] = 0 ;
      b[i] = a[i] ;
    }

    // variables containing local array elements of distributed arrays 

    float [] a_block = a.dat() ;
    float [] b_block = b.dat() ;
    float [] f_block = f.dat() ;

    for(int s = 0 ; s < P ; s++) {

      for(int i = 0 ; i < B ; i++)  // B : local block size
        for(int j = 0 ; j < B ; j++) 
          f_block[i] += force(a_block[i], b_block[j]) ;


      // cyclically shift `b_block' (by amount B in x dim)...

      MPI.COMM_WORLD.Sendrecv_replace(b_block, 0, B, MPI.FLOAT, 
                                      right, 99, left, 99) ;

    }                 

  }
\end{verbatim}
\Line 
\normalsize
\caption{Data parallel version of the ``N-body'' example with MPI 
         communications.\label{fig:data-MPI}}
\end{figure}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Discussion}

In this chapter we discussed motivations for introducing an HPspmd
programming model: a SPMD framework for using libraries based on
distributed arrays.  It adopts the model of distributed arrays
standardized by the HPF Forum, but relinquishes the high-level
single-threaded model of the HPF language.  This makes compilers or
translators for the HPspmd-extended languages a relatively
straightforward proposition.  As a concrete example, we described the
specific syntax of HPJava.

Two recent languages that have some similarities to our HPspmd
languages are F-\,- and ZPL.  F-\,- \cite{FMM} is an extended Fortran
dialect for SPMD programming.  The approach is different to the one
proposed here.  There is no analogue of global subscripts, or HPF-like
distribution formats.  In F-\,- the logical model of communication is
built into the language---remote memory access with intrinsics for
synchronization---where our basic philosophy is to provide
communication through separate libraries.  ZPL \cite{ZPL} is a
array parallel programming language for scientific computations.
It has a construct for performing computations over a {\em region}, or
set of indices, quite similar to our {\em overall} construct.
Communication is more explicit than HPF, but not as
explicit as in the language discussed in this article.

At the time of writing the HPJava translator is partially operational.
%Currently, we have had quite successful running HPJava code calling mpiJava
%introduced in previous section.  
All examples given in this chapter can be compiled and run.
Ongoing work will complete the functionality, and add some optimization
for the generated code.  The language definition calls for full
compile-time or runtime checking of the constraints on locality of
reference.  The translator will be enhanced to add these.  

In the next chapter we will further discuss the mpiJava library
briefly mentioned in the previous section. 