
\documentclass{book}

%\usepackage{psfig}

%\usepackage[pdftex]{graphicx}
%\usepackage{epstopdf}
\usepackage{graphicx}


\newenvironment{code}{\begin{minipage}[t]{\linewidth}\small}{\end{minipage}}

    % "code" environment always used inside a "displaymath" environment.

    % For unknown reasons, including displaymath as part of the "code"
    % definition causes unwanted space at the beginning of the following line.

\newenvironment{bigCode}{\begin{minipage}[t]{\linewidth}}{\end{minipage}}


\newtheorem{usageRule}{Rule}

\newtheorem{recommendation}{Recommendation}


%\newcommand{\trans}[1]{{\bf T}\left[{#1}\right]}
\newcommand{\trans}[2]{{\bf T}\left[{#1}\left|{#2}\right.\right]}

\newcommand{\transp}[1]{{\bf T}_{\mbox{\small pre}}\left[{#1}\right]}
%\newcommand{\transv}[1]{{\bf T}_{\mbox{\small val}}\left[{#1}\right]}
\newcommand{\transv}[1]{{\bf T}\left[{#1}\right]}

\newcommand{\transpV}[1]{{\bf T}^V_{\mbox{\small pre}}\left[{#1}\right]}
\newcommand{\transvV}[1]{{\bf T}^V_{\mbox{\small val}}\left[{#1}\right]}

\newcommand{\transd}[1]{{\bf T}_{\mbox{\small dat}}\left[{#1}\right]}
\newcommand{\transb}[1]{{\bf T}_{\mbox{\small bas}}\left[{#1}\right]}
\newcommand{\transg}[1]{{\bf T}_{\mbox{\small grp}}\left[{#1}\right]}
\newcommand{\transr}[2]{{\bf T}_{#1}\left[{#2}\right]}

\newcommand{\transdV}[1]{{\bf T}^V_{\mbox{\small dat}}\left[{#1}\right]}
\newcommand{\transbV}[1]{{\bf T}^V_{\mbox{\small bas}}\left[{#1}\right]}
\newcommand{\transgV}[1]{{\bf T}^V_{\mbox{\small grp}}\left[{#1}\right]}
\newcommand{\transrV}[2]{{\bf T}^V_{#1}\left[{#2}\right]}

\newcommand{\egrp}{e_{\mbox{\small grp}}}
\newcommand{\erng}{e_{\mbox{\small rng}}}
%\newcommand{\eglb}{e_{\mbox{\small glb}}}
\newcommand{\elo}{e_{\mbox{\small lo}}}
\newcommand{\ehi}{e_{\mbox{\small hi}}}
\newcommand{\estp}{e_{\mbox{\small stp}}}
\newcommand{\eobj}{e_{\mbox{\small obj}}}
%\newcommand{\earr}{e_{\mbox{\small arr}}}
\newcommand{\eloc}{e_{\mbox{\small loc}}}
\newcommand{\ebool}{e_{\mbox{\small bool}}}
\newcommand{\eshf}{e_{\mbox{\small shf}}}


\newcommand{\noDec}[1]{{\it NODEC}\left({#1}\right)}


\begin{document}

\bibliographystyle{plain}

%\pagestyle{plain}

\title{Parallel Programming in HPJava}

\author{Bryan Carpenter, Guansong Zhang, Han-Ku Lee, Sang Boem Lim, \ldots\\
        \\
        School of Computational Science and Information Technology \\
        Florida State Universtiy \\
        Tallahassee, FL 32306-4120}

\maketitle

\newpage

\tableofcontents

\chapter{Introduction}

HPJava is a language for parallel programming.  It extends the Java
language with some syntax for manipulating a new kind of parallel data
structure---the {\em distributed array}.  The specific extensions
evolved out of work on Fortran 90 and High Performance Fortran (HPF),
but the programming model of HPJava is different to
HPF.  The HPJava model is one of explicitly cooperating
processes.  It is an implementation of of the {\em Single Program,
Multiple Data} (SPMD) model.  All processes execute the same
program, but the components of data structures---in our case the
elements of distributed arrays---are divided across processes.
Individual processes operate on the locally owned segment of an entire
array.  At some points in the computation processes usually need access to
elements owned by their peers.  Explicit communications are needed to
permit this access.

This general scheme has been very successful in realistic programs.  It
is probably no exaggeration to say that most successful
applications of parallel computing to large scientific and numerical
problems are programmed in this style.  So HPJava is attempting to
add some extra support at the language level for established
practises of programmers.

{\em [HPJava is {\em not} a parallelizing compiler.  Before writing
an HPJava program you must understand parallel algorithms.  HPJava
is a notation for expressing and implementing parallel algorithms.]}


\chapter{Processes and Arrays}

\section{Processes and Process Grids\label{sec:processes}}

An HPJava program is started concurrently in all members of some
process set%
\footnote{Most of the time we will talk about {\em processes} rather than
processors.  The different processes may be
running on different {\em processors} to achieve a parallel speedup.}.
From the point of view of the HPJava program, the processes are
organized and identified through special objects representing {\em process
groups}.  In general the processes in an HPJava group are arranged in
multidimensional grids.

Suppose a program is running concurrently on 6 or
more processes.  It may then define a 2 by 3 process grid as follows
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs2 p = new Procs2(2, 3) ;
\end{verbatim}
\end{code}
\end{displaymath}
{\tt Procs2} is a class describing 2-dimensional grids of processes.
The grid {\tt p} is visualized in Figure \ref{fig:procs2}.
This figure assumes that the program was executing in 11 processes.
The call to the {\tt Procs2} constructor selected 6 of these available
processes and incorporated them into a grid.

\begin{figure*}[tbp]
%\centerline{\psfig{figure=procs2.eps,width=3in}}
\centerline{\includegraphics[width=3in]{procs2.eps}}
\caption{The process grid represented by {\tt p}.\label{fig:procs2}}
\end{figure*}

The {\tt Procs2} constructor is an example of a {\em collective
operation}.  It should be invoked concurrently by all members
of the active process group.

{\tt Procs2} is a subclass of the special class {\tt Group}.
The {\tt Group} class has a privileged status in the HPJava language.
An object that inherits this class can be used in various special places.
For example, it can be used to parametrize an {\em on construct}.

After creating {\tt p} we will probably
want to perform some operations within the processes of the grid.
An {\tt on} construct restricts control to processes in its
parameter group.  So in
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs2 p = new Procs2(2, 3) ;
  on(p) {
    ... body of on construct ...
  }
\end{verbatim}
\end{code}
\end{displaymath}
the code inside the construct is only executed by processes belonging
to {\tt p}.
In Figure \ref{fig:procs2}, the five processors outside the grid would
skip this block of code.

This {\tt on} construct establishes {\tt p} as the {\em active process group}
within its body.  In several of the operations introduced later, the
current active process group will act as a ``default'' group
parameter.

An auxilliary class called {\tt Dimension} is associated with
process grids.  Objects of this class
describe a particular dimension or axis of a particular process grid.
They will are called {\em process dimensions}.  The process
dimensions of a grid are available through the inquiry method
$\mbox{\tt dim(}r\mbox{\tt )}$.
The argument $r$ is in the range $0, \ldots, R - 1$,
where in general $R$ is the rank (dimensionality) of the grid.

The {\tt Dimension} class in turn has a method {\tt crd}, which
returns the local {\em process coordinate} associated with the
dimension---ie, the position of the local process within the dimension.
If we executed the following HPJava program
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs2 p = new Procs2(2, 3) ;
  on(p) {
    Dimension d = p.dim(0), e = p.dim(1) ;

    System.out.println("My coordinates are (" + d.crd() +
                                         ", " + e.crd() + ")") ;
  }
\end{verbatim}
\end{code}
\end{displaymath}
we might see the output
\begin{displaymath}
\begin{code}
\begin{verbatim}
  My coordinates are (0, 2)
  My coordinates are (1, 2)
  My coordinates are (0, 0)
  My coordinates are (1, 0)
  My coordinates are (1, 1)
  My coordinates are (0, 1)
\end{verbatim}
\end{code}
\end{displaymath}
The dimensions of {\tt p} are illustrated in Figure \ref{fig:procs2dims}.
Because the 6 processes are running concurrently there is no way to
predict the order in which the messages appear (and if we were unlucky
they might even be interleaved).  If we
applied {\tt crd()} to {\tt d} or {\tt e} in a process outside {\tt p}
(such as one of the shaded processes in Figure \ref{fig:procs2}) we
should expect an exception.

\begin{figure*}[tbp]
%\centerline{\psfig{figure=procs2dims,width=2.5in}}
\centerline{\includegraphics[width=2.5in]{procs2dims.eps}}
\caption{The process dimensions and coordinates in {\tt
p}.\label{fig:procs2dims}}
\end{figure*}

There is nothing
special about 2-dimensional grids.  The full {\tt Group} hierarchy of
HPJava includes the classes of Figure \ref{fig:procs}.  Not
surprisingly, {\tt Procs1} is a one-dimensional process ``grid''.  Less
obviously, {\tt Procs0} is a group containing exactly one process.
Higher dimensional grids are also allowed.

\begin{figure*}[tbp]
%\centerline{\psfig{figure=procs.eps,width=3in}}
\centerline{\includegraphics[width=3in]{procs.eps}}
\caption{The {\tt Group} hierachy of HPJava.\label{fig:procs}}
\end{figure*}

Note that so far the only special {\em syntax} we have added to the
Java language is the {\tt on} construct.  The {\tt Group}
class has special status in HPJava\footnote{Much as {\tt Throwable} has a
special status in normal Java---it is the only class that can parametrize
a {\tt catch} clause.}, but syntactically it is just a class.

\section{Distributed Arrays\label{sec:arrays}}

The biggest new feature HPJava adds to Java is the {\em distributed
array}.
A distributed array is a {\em collective object} shared by a number
of processes.  Like an ordinary array, a distributed array has some
index space and stores a collection of elements of fixed type.  Unlike
an ordinary array, the index space and associated elements are
scattered across the processes that share the array.

There are some similarities between the way HPJava
distributed arrays are used and the way ordinary Java arrays are used.
There are also a number of differences.  Apart from the fact that
their elements are distributed in the manner just mentioned, the new
HPJava arrays are true multi-dimensional arrays like those of Fortran
(or, for that matter, C).  As in Fortran, it is possible to form a {\em
regular section} of an array.  These characteristics of Fortran arrays
have evolved to support scientific and parallel algorithms, and we
consider them to be very desirable.

Bearing in mind these distinctions it does not seem a good idea
to try and force the new HPJava arrays into a syntax directly
reminiscent of ordinary Java arrays.  Instead, HPJava distributed
arrays look and feel quite different from standard arrays%
%\footnote{It is arguable that this is one of the
%places where High Performance Fortran went wrong.  HPF tries to make
%distributed arrays semantically indistinguishable from the sequential
%arrays of the base language (Fortran).  This means on the one hand
%that distributed arrays have to allow the same kind of unlimiteid random
%access as sequential arrays, which is difficult and inefficient to
%support.  On the other hand, there are places where it is difficult
%to be sure at compile-time whether a subprogram will be
%dealing with a sequential array or a distributed array.  The compiler
%may have to make a worst case assumption, leading to inefficiencies in
%treatment of sequential arrays.}%
.  Of course HPJava includes Java as a
subset, and ordinary Java arrays can and should be used freely in an
HPJava program.  But they do not have a close relationship to the new
distributed arrays.
%The distributed array types can be regarded as a
%series of specialized container classes, dressed up in a special
%syntax.

The type signatures and constructors of distributed arrays use double
brackets to distinguish them from ordinary Java arrays.  The
distribution of the index space is parametrized by objects belonging
to another class that has a special status in HPJava: the {\tt Range} class.
In the following example we create a two-dimensional, {\tt N} by
{\tt N}, array of floating
point numbers, with elements distributed over the grid {\tt p}.
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs2 p = new Procs2(2, 3) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;
    Range y = new BlockRange(N, p.dim(1)) ;

    float [[-,-]] a = new float [[x, y]] ;

    ...
  }
\end{verbatim}
\end{code}
\end{displaymath}
The decomposition of this array for the case {\tt N} = 8 is illustrated
in Figure \ref{fig:array}.  The choice of subclass {\tt BlockRange} for
the index ranges of the array means that the index space of each array dimension
is broken down into consecutive blocks.  Other possible distribution
formats will be discussed later.  Notice how process dimensions are
passed as arguments to the range constructors, specifying which
dimensions the range is to be distributed over.  Ranges of a single
array must be distributed over different dimensions of the same process
grid%
\footnote{Unless they are collapsed; see section \ref{sec:collapsed}.}%
.

\begin{figure*}[tbp]
%\centerline{\psfig{figure=array.eps,width=4.5in}}
\centerline{\includegraphics[width=4.5in]{array.eps}}
\caption{A two-dimensional array distributed over {\tt p}.\label{fig:array}}
\end{figure*}

\section{Parallel Programming\label{sec:parallel}}

The previous section included a simple example of how to create a distributed
array.  How do we use this kind of array?  Figure \ref{fig:addition} gives
an example---parallel addition of two matrices.

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs2 p = new Procs2(P, P) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;
    Range y = new BlockRange(N, p.dim(1)) ;

    float [[-,-]] a = new float [[x, y]], b = new float [[x, y]],
                  c = new float [[x, y]] ;

    ... initialize values in `a', `b'

    overall(i = x for :)
      overall(j = y for :)
        c [i, j] = a [i, j] + b [i, j] ;
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{A parallel matrix addition.\label{fig:addition}}
\end{figure*}

The {\em overall construct} is the second special control construct
of HPJava.  It implements a parallel loop,
sharing a heritage with the {\em forall} construct of HPF.  The
colon in the {\tt overall} headers of the example is shorthand for an
index {\em triplet}%
\footnote{The syntax for triplets is lifted directly from Fortran 90.}%
.  In general a triplet can include a lower bound, an
upper bound, and a step, as follows:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    overall(i = x for l : u : s)
\end{verbatim}
\end{code}
\end{displaymath}
In general $l$, $u$ and $s$ can be any integer expressions.
The third member of the triplet (the step) is optional.  The
default step is 1, and most often we see something like
\begin{displaymath}
\begin{code}
\begin{verbatim}
    overall(i = x for l : u)
\end{verbatim}
\end{code}
\end{displaymath}
Finally, either or both of the expressions $l$ and $u$ can be omitted.
The lower bound defaults to 0 and the upper bound defaults to
$N - 1$, where $N$ is the extent of the range appearing before the
{\tt for} keyword.  So in the original example the line
\begin{displaymath}
\begin{code}
\begin{verbatim}
    overall(i = x for :)
\end{verbatim}
\end{code}
\end{displaymath}
means ``repeat for all elements, $i$, of the range $x$'', and is equivalent to
\begin{displaymath}
\begin{code}
\begin{verbatim}
    overall(i = x for 0 : x.size() - 1 : 1)
\end{verbatim}
\end{code}
\end{displaymath}
The {\tt size} member of {\tt Range} returns the extent of the range.

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs2 p = new Procs2(P, P) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;
    Range y = new BlockRange(N, p.dim(1)) ;

    float [[-,-]] a = new float [[x, y]] ;

    float [[-,-]] n = new float [[x, y]], s = new float [[x, y]],
                  e = new float [[x, y]], w = new float [[x, y]] ;

    ... initialize `a'

    Adlib.shift(s, a,  1, 0) ;
    Adlib.shift(n, a, -1, 0) ;
    Adlib.shift(w, a,  1, 1) ;
    Adlib.shift(e, a, -1, 1) ;

    overall(i = x for 1 : N - 2)
      overall(j = y for 1 : N - 2)
        a [i, j] = 0.25 * (n [i, j] + s [i, j] + e [i, j] + w [i, j]) ;
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{A parallel stencil update program.\label{fig:stencil}}
\end{figure*}

For readers familiar with the {\tt forall} construct of HPF (or Fortran
95) the only unexpected part of the {\tt overall} syntax is
the reference to a range object in front of the triplet.  The
significance of this will be discussed at length in the next section.
Meanwhile we give
another example of a parallel program.  Figure \ref{fig:stencil} is a
simple example of a ``stencil update''.  Each interior element of array {\tt a}
is supposed to be replaced by the average of the values of its
neighbours:
\begin{displaymath}
  a [i, j] \,\, \leftarrow \,\,
(a [i - 1, j] + a [i + 1, j] + a [i, j - 1] + a [i, j + 1]) / 4
\end{displaymath}

The {\tt shift} operation is not a new feature of the HPJava
{\em language}, as such.  Instead it is a member of a particular library
called {\em Adlib}.
This is a library of collective operations on distributed arrays%
\footnote{Many of them are modelled on the
array transformational intrinsic functions of Fortran 90.}%
.  The function {\tt shift} is overloaded to apply to various kinds of
array.  In this example we are using the instance applicable to
two dimensional arrays with {\tt float} elements:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    void shift(float [[-,-]] destination, float [[-,-]] source,
               int shiftAmount, int dimension) ;
\end{verbatim}
\end{code}
\end{displaymath}
The array arguments should have the same shape and distribution format.
The values in the {\tt source} array are copied to the {\tt destination}
array, shifted by {\tt shiftAmount} in the {\tt dimension}'th array
dimension%
\footnote{Edge values from {\tt source} that have no target in {\tt
destination} are discarded; edge elements of {\tt destination} that are
not overwritten by elements from {\tt source} are unchanged from their
input value.}%
.

In the example program, arrays of North, South, East and West neighbours
are created using {\tt shift}, then they are averaged in {\tt overall} loops.
An obvious question is: why go to the trouble of setting up
these arrays?  Surely it would be easier to write directly:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    overall(i = x for 1 : N - 2)
      overall(j = y for 1 : N - 2)
        a [i, j] = 0.25 * (a [i - 1, j] + a [i + 1, j] +
                           a [i, j - 1] + a [i, j + 1]) ;
\end{verbatim}
\end{code}
\end{displaymath}
The answer relates to the nature of the symbols {\tt i} and {\tt j}.
No declaration was given for these names, but it would be reasonable
to assume that they stand for integer values.  

They don't.

\section{Locations}
\label{sec:locations}

An HPJava range object can be thought of as a set of abstract {\em locations}.
In our earlier example,
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs2 p = new Procs2(2, 3) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;
    Range y = new BlockRange(N, p.dim(1)) ;

    ...
  }
\end{verbatim}
\end{code}
\end{displaymath}
the range {\tt x}, for example, contains {\tt N} locations.
In an {\tt overall} construct such as
\begin{displaymath}
\begin{code}
\begin{verbatim}
    overall(i = x for 1 : N - 2) {
      ...
    }
\end{verbatim}
\end{code}
\end{displaymath}
the symbol {\tt i} is called a {\em distributed index}.
The value associated with a distributed index is a {\em location}, {\em
not} an integer value.

With a few exceptions that will be discussed later, the
subscript of a distributed array {\em must be a distributed index}, and the
location must be an element of the range associated with the array dimension.
This is why we introduced the temporary arrays for neighbours in the
stencil update example of the previous section.  Arbitrary expressions
are not usually legal subscripts for a distributed array.
If we want to combine elements of
arrays that are not precisely aligned, we first have to use a library
function such as {\tt shift} to align them%
%\footnote{Because stencil updates are so common, we will see in section
%\ref{sec:ghostRegions} that, as a special case, the language {\em
%does} allow constantly shifted locations as subscripts.  But exploiting
%this facility needs some special provisions.}%
.

Figure \ref{fig:locations} is an attempt to visualize the mapping of
locations from {\tt x} and {\tt y}.  We will write the locations of
{\tt x} as {\tt x[0]}, {\tt x[1]}, \ldots, {\tt x[N - 1]}.  Each
location is mapped to a particular group of processes.  Location {\tt
x[1]} is mapped to the three processes with coordinates $(0,0)$,
$(0,1)$ and $(0,2)$.  Location {\tt y[4]} is mapped to the two
processes with coordinates $(0,1)$ and $(1,1)$.

\begin{figure*}[tbp]
%\centerline{\psfig{figure=locations.eps,width=4in}}
\centerline{\includegraphics[width=4in]{locations.eps}}
\caption{Mapping of {\tt x} and {\tt y} locations to the grid {\tt p}.%
\label{fig:locations}}
\end{figure*}

Besides {\tt overall}, there is another control construct in HPJava that
defines a distributed index---the simpler {\em at construct}.  Suppose we
want to update or access a single element of a distributed array
(rather than accessing a whole set of of elements in parallel).
it is not allowed to write simply
\begin{displaymath}
\begin{code}
\begin{verbatim}
    float [[-,-]] a = new float [[x, y]] ;
    ...
    a [1, 4] = 73 ;
\end{verbatim}
\end{code}
\end{displaymath}
because 1 and 4 are not distributed indices, and therefore not legal
subscripts.  We {\em can} write:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    float [[-,-]] a = new float [[x, y]] ;
    ...
    at(i = x [1])
      at(j = y [4])
        a [i, j] = 73 ;
\end{verbatim}
\end{code}
\end{displaymath}
The symbols {\tt i} and {\tt j}, scoped within the construct
bodies, are distributed indices.

The operational meaning of the {\tt at} construct should be fairly
clear.  It is similar to the {\tt on} construct.  It restricts control
to processes in the set that hold the specified location.
Referring again to Figure \ref{fig:locations}, the outer
\begin{displaymath}
\begin{code}
\begin{verbatim}
    at(i = x [1])
\end{verbatim}
\end{code}
\end{displaymath}
construct limits execution of its body to processes with coordinates
$(0,0)$, $(0,1)$ and $(0,2)$.  The inner
\begin{displaymath}
\begin{code}
\begin{verbatim}
    at(j = y [4])
\end{verbatim}
\end{code}
\end{displaymath}
then restricts execution down to just process $(0,1)$.  This is exactly
the process that owns element {\tt a[1,4]}.  The odd restriction that
subscripts must be distributed indices helps ensure that processes only
manipulate array elements stored locally.  If a process has to
access non-local data, some explicit library call is needed to fetch
it.

An operational definition of {\tt overall} can be given in terms
of the simpler {\tt at} construct.  If {\tt s} is greater than zero, the construct
\begin{displaymath}
\begin{code}
\begin{verbatim}
    overall(i = x for l : u : s) {
      ...
    }
\end{verbatim}
\end{code}
\end{displaymath}
is equivalent in behaviour to
\begin{displaymath}
\begin{code}
\begin{verbatim}
    for(int n = l; n <= u ; n += s)
      at(i = x [n]) {
        ...
      }
\end{verbatim}
\end{code}
\end{displaymath}
If {\tt s} is less than zero, it is equivalent to
\begin{displaymath}
\begin{code}
\begin{verbatim}
    for(int n = l; n >= u ; n += s)
      at(i = x [n]) {
        ...
      }
\end{verbatim}
\end{code}
\end{displaymath}
The bodies of the {\tt at} constructs are skipped for values of {\tt n}
that don't correspond to locally held elements.  In practise an HPJava
compiler can translate the {\tt overall} construct much more efficiently
than this definition suggests.

The {\tt at} construct completes the contingent of new control
constructs in HPJava.  We sometimes refer to the three constructs
{\tt on}, {\tt at} and {\tt overall} as {\em distributed control}
constructs (and sometimes, more grandiosely, as {\em structured SPMD}
control constructs).

The backquote symbol, \verb$`$, can be used as a postfix operator on
a distributed index, thus:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    i`
\end{verbatim}
\end{code}
\end{displaymath}
This expression is read ``i-primed'', and evaluates to the integer global
index value.  In the operational definition of the {\tt overall} given above,
this is the value called {\tt n}.

We now know enough about HPJava to write some more complete examples.

\section{A Complete Example}

The example of Figure \ref{fig:jacobi} only uses language features
introduced in the preceding sections.  It introduces two new
library functions.

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs2 p = new Procs2(P, P) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;
    Range y = new BlockRange(N, p.dim(1)) ;

    float [[-,-]] a = new float [[x, y]] ;

    // Initialize `a': set boundary values.

    overall(i = x for :)
      overall(j = y for :)
        if(i` == 0 || i` == N - 1 || j` == 0 || j` == N - 1)
          a [i, j] = i` * i` - j` * j` ;
        else
          a [i, j] = 0.0 ;
    
    // Main loop.

    float [[-,-]] n = new float [[x, y]], s = new float [[x, y]],
                  e = new float [[x, y]], w = new float [[x, y]] ;

    float [[-,-]] r = new float [[x, y]] ;

    do {
      Adlib.shift(n, a,  1, 0) ;
      Adlib.shift(s, a, -1, 0) ;
      Adlib.shift(e, a,  1, 1) ;
      Adlib.shift(w, a, -1, 1) ;

      overall(i = x for 1 : N - 2)
        overall(j = y for 1 : N - 2) {
          float newA ;

          newA = 0.25 * (n [i, j] + s [i, j] + e [i, j] + w [i, j]) ;
          r [i, j] = Math.abs(newA - a [i, j]) ;
          a [i, j] = newA ;
        }

    } while(Adlib.maxval(r) > EPS) ;

    // Output results.

    Adlib.printArray(a) ;
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Solution of Laplace equation by Jacobi relaxation.\label{fig:jacobi}}
\end{figure*}

The problem is a very well-known one: solution of the two-dimensional
Laplace equation with Dirichlet boundary conditions by the iterative
Jacobi relaxation method%
\footnote{Maybe we could have chosen a more creative first example.
But the point is to explain language features---the more familiar the
algorithm, the better.}%
.
The boundary conditions of the equation are fixed by setting edge
elements of an array.  These elements don't change throughout the
computation.  The solution for the interior region is obtained by
iteration from some arbitrary starting values (zero, here).  A single
iteration involves replacing each interior element by the average of
its neighbouring values.  A similar update was already discussed in
section \ref{sec:parallel}.  Here we put it in the context of a working
program.

The initialization is done with a pair of nested {\tt overall} constructs.
Inside, a conditional tests if we are on an edge of the array.  If
so, the element values are set to some chosen expression---the
boundary function.  Otherwise we zero an interior element.
As discussed at the end of the last section we, apply the operator
\verb$`$ to distributed indices to get the global loop index.

Notice that one can freely use ordinary Java constructs like if inside
an {\tt overall} construct.  HPJava distributed control construct are true,
compositional control constructs.  They differ in this respect from the
HPF forall construct, which has restrictive rules about what kind of
statement can appear in its body.

If preferred, the edges could have been initialized using {\tt at}
constructs:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    at(i = x [0])
      overall(j = y for :)
        a [i, j] = i` * i` - j` * j` ;
    
    at(i = x [N - 1])
      overall(j = y for :)
        a [i, j] = i` * i` - j` * j` ;
    
    at(j = y [0])
      overall(i = x for :)
        a [i, j] = i` * i` - j` * j` ;
    
    at(j = y [N - 1])
      overall(i = x for :)
        a [i, j] = i` * i` - j` * j` ;
\end{verbatim}
\end{code}
\end{displaymath}
with the interior initialized separately using nested {\tt overall} constructs:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    overall(i = x for 1 : N - 2)
      overall(j = y for 1 : N - 2)
        a [i, j] = 0.0
\end{verbatim}
\end{code}
\end{displaymath}
This version is more long-winded but potentially more efficient,
because it simplifies the inner loop bodies, improving the scope for
compiler optimization.

The body of the main loop contains {\tt shift} operations and nested
{\tt overall} loops.  The body of the inner loop is slightly more complicated
than the version in figure \ref{fig:stencil} because it saves changes
to the main array in a separate array {\tt r}.

Note the declaration of the {\tt float} temporary {\tt newA} inside the
body of the parallel loop.  This is perfectly good practise.  The
temporary is just an ordinary scalar Java variable---the HPJava
translator doesn't treat it specially.  Also note the call to a Java
library function {\tt abs} inside the loop.  As we have emphasized, any
normal Java operation is allowed inside an HPJava distributed control
construct.

The main loop terminates when the largest change in any element is
smaller than some predefined value {\tt EPS}.  The collective library
function {\tt maxval} finds the largest element of distributed array,
and broadcasts its value to all processes that call the function.
{\em [Need to initialize edges of {\tt r} to zero.]}

Finally a collective library function {\tt printArray} prints a formatted
versions of the array on the standard output stream.

\chapter{More on Mapping Arrays}

\section{Other Distribution Formats\label{sec:otherDistributions}}

HPJava follows HPF in allowing several different distribution formats
for the dimensions of its distributed arrays.  The new formats are
provided without further extension to the syntax of the language.
Instead the {\tt Range} class hierarchy is extended.  The full hierarchy
is shown in Figure \ref{fig:ranges}.

\begin{figure*}[tbp]
%\centerline{\psfig{figure=ranges.eps,width=2in}}
\centerline{\includegraphics[width=2in]{ranges.eps}}
\caption{The {\tt Range} hierachy of HPJava.\label{fig:ranges}}
\end{figure*}

The {\tt BlockRange} subclass is very familiar by now.  The
{\tt Dimension} class is also familiar, although previously it wasn't
presented as a range class---later examples will demonstrate how
it can be convenient to use process dimensions as array ranges.
{\tt CyclicRange} and {\tt BlockCyclicRange} are directly analogous
with the cyclic and block-cyclic distribution formats available in
HPF.

Cyclic distributions are sometimes favoured because they can lead to
better {\em load balancing} than the simple block distributions
introduced so far.  Some algorithms (for example dense matrix
algorithms) don't have the kind of locality that favours block
distribution for stencil updates, but they do involve phases where
parallel computations are limited to subsections of the whole
array.  In block distributions these sections may map to
only a fraction of the available processes, leaving the remaining
processes idle.  Here is a contrived example
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs2 p = new Procs2(2, 3) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;
    Range y = new BlockRange(N, p.dim(1)) ;

    float [[-,-]] a = new float [[x, y]] ;

    overall(i = x for 0 : N / 2 - 1)
      overall(j = y for 0 : N / 2 - 1)
        a [i, j] = complicatedFunction(i`, j`) ;
  }
\end{verbatim}
\end{code}
\end{displaymath}
The point here is that the {\tt overall} constructs only traverse half the
ranges of the array they process.  As shown in Figure \ref{fig:blockBalance},
this leads to a very poor distribution of workload.  The process with
coordinates $(0,0)$ does nearly all the work.  The process at $(0,1)$
has a few elements to work on, and all the other processes are idle.

\begin{figure*}[tbp]
%\centerline{\psfig{figure=blockBalance.eps,width=4in}}
\centerline{\includegraphics[width=4in]{blockBalance.eps}}
\caption{Work distribution for an example with block
distribution.\label{fig:blockBalance}}
\end{figure*}

In {\em cyclic} distribution format, the
index space is mapped to the process dimension in wraparound
fashion.  If we change our example to
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs2 p = new Procs2(2, 3) ;
  on(p) {
    Range x = new CyclicRange(N, p.dim(0)) ;
    Range y = new CyclicRange(N, p.dim(1)) ;

    float [[-,-]] a = new float [[x, y]] ;

    overall(i = x for 0 : N / 2 - 1)
      overall(j = y for 0 : N / 2 - 1)
        a [i, j] = complicatedFunction(i`, j`) ;
  }
\end{verbatim}
\end{code}
\end{displaymath}
Figure \ref{fig:cyclicBalance} shows that the
imbalance is not nearly as extreme.
Notice that nothing changed in the program apart from the choice of
range constructors.  This is an attractive feature that HPJava shares
with HPF.  If HPJava programs are written in a sufficiently pure data
parallel style (using {\tt overall} loops and collective array operations) it
is often possible to change the distribution format of arrays
dramatically while leaving much of the code that processes them
unchanged.  HPJava does not guarantee this property in the same way
that HPF does, and fully optimized HPJava programs are unlikely to be
so easily redistributed.  But it is still a useful property.

\begin{figure*}[tbp]
%\centerline{\psfig{figure=cyclicBalance.eps,width=4in}}
\centerline{\includegraphics[width=4in]{cyclicBalance.eps}}
\caption{Work distribution for an example with cyclic
distribution.\label{fig:cyclicBalance}}
\end{figure*}

As a more graphic example of how cyclic distribution
can improve load balancing, consider the Mandelbrot set code of Figure
\ref{fig:mandel}.  Points away from the set are generally eliminated in
a few iterations, whereas those close to the set or inside it take much
more computation---points are assumed to be inside the set, and the
corresponding element of the array is set to 1, when the number of
iterations reaches {\tt CUTOFF}.  In the case {\tt N} = 64 (with {\tt
CUTOFF} = 100), the block decomposition of the set is shown in Figure
\ref{fig:blockMandelbrot}.  The middle column of processors will do
most of the work, because they hold most of the set.  Changing the
class of the ranges to {\tt CyclicRange} gives the much more even
distribution shown in Figure \ref{fig:cyclicMandelbrot}.

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs2 p = new Procs2(2, 3) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;
    Range y = new BlockRange(N, p.dim(1)) ;

    int [[-,-]] set = new int [[x, y]] ;

    overall(i = x for :)
      overall(j = y for :) {
        float cr = (4.0 * i` - 2 * N) / N ;
        float ci = (4.0 * j` - 2 * N) / N ;

        float zr = cr, zi = ci ;

        set [i, j] = 0 ;

        int k = 0 ;
        while (zr * zr + zi * zi < 4.0) {
          if(k++ == CUTOFF) {
            set [i, j] = 1 ;
            break ;
          } 

          // z = c + z * z

          float newr = cr + zr * zr - zi * zi ;
          float newi = ci + 2 * zr * zi ;

          zr = newr ;
          zi = newi ;
        }

      }

    Adlib.printArray(set) ;
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Mandelbrot set computation.\label{fig:mandel}}
\end{figure*}

\begin{figure*}[tbp]
%\centerline{\psfig{figure=blockMandelbrot.eps,width=3in}}
\centerline{\includegraphics[width=3in]{blockMandelbrot.eps}}
\caption{Blockwise decomposition of the Mandelbrot set (black
region).\label{fig:blockMandelbrot}}
\end{figure*}

\begin{figure*}[tbp]
%\centerline{\psfig{figure=cyclicMandelbrot.eps,width=3in}}
\centerline{\includegraphics[width=3in]{cyclicMandelbrot.eps}}
\caption{Cyclic decomposition of the Mandelbrot set.%
\label{fig:cyclicMandelbrot}}
\end{figure*}



Block cyclic distribution format is a generalization of cyclic
distribution that is used in some libraries for parallel
linear algebra%
\footnote{Notably ScaLAPACK.}%
.  It will not be discussed further here.

The {\tt ExtBlockRange} subclass represents block-distributed ranges
extended with {\em ghost regions}.  

\clearpage

\section{Ghost Regions\label{sec:ghostRegions}}

In a distributed array with ghost
regions, the memory for the locally held block of elements is allocated
with extra space around the edges.  These extra locations can be used
to cache some of the element values properly belonging to adjacent
processors.  The inner loop of algorithms involving stencil updates can
then written very simply.  In accessing neighbouring elements, the edges
of the block don't need special treatment.  Rather than throwing an
exception for an out of range subscript, shifted indices find the
proper values cached in the ghost region.  This is such an important
technique in real codes that HPJava has a special extension to make
it possible.

This is one place where the rule that the subscript of a distributed
array must be a distributed index is relaxed.  
In a special, slightly
idiosyncratic, piece of syntax, the following expression
\begin{displaymath}
\mbox{\em name}  \pm \mbox{\em expression}
\end{displaymath}
is a legal subscript
if {\em name} is a distributed index and {\em expression}
is an integer expression---in practice usually a small
constant.  This is called a {\em shifted index}.  The significance of the
shifted index is that an element
displaced from the original location will be accessed.  If the shift
puts the location outside the local block {\em plus} surrounding ghost region,
an exception will occur.
Using this syntax the example at the end of section \ref{sec:arrays}:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    overall(i = x for 1 : N - 2)
      overall(j = y for 1 : N - 2)
        a [i, j] = 0.25 * (a [i - 1, j] + a [i + 1, j] +
                           a [i, j - 1] + a [i, j + 1]) ;
\end{verbatim}
\end{code}
\end{displaymath}
{\em is} allowed {\em if} the array {\tt a} has suitable ghost extensions%
\footnote{We need to be rather clear about the semantics of this extension,
because it is an odd case.
Suppose {\tt x} is the range associated with the distributed index {\tt i}.
Let {\tt j} be the location {\tt x [i` + e]}.  If {\tt j} is mapped to
the same processes as {\tt i}, and the shifted index {\tt i + e} is used
as a subscript, it behaves just like the location {\tt j}.  If {\tt j}
is mapped to a different set of processes from {\tt i}, and {\tt i + e}
is used as a subscript, the resulting reference must be to an element
in a ghost region on the process holding {\tt i}.}%
.

Ghost regions are not magic.  The values cached around the edges of a
local block can only be made consistent with the values held in blocks on
adjacent processes by a suitable communication.  A library function
called {\tt writeHalo} updates the cached values in the ghost regions
with proper element values from neigbouring processes.

Figure \ref{fig:ghostJacobi} is a version of the Laplace program that
uses ghost regions.  The omitted code is unchanged
from Figure \ref{fig:jacobi}.  The last two arguments of the
{\tt ExtBlockRange} constructor define the widths of the ghost
regions added to the bottom and top (respectively) of the local
block.  

In this version we still 
introduced one temporary array, called {\tt b}.  The reason is
is that in Jacobi relaxation one is supposed to express all the new
values in terms of values from the previous iteration.
The library function {\tt copy} copies elements between two
{\em aligned} arrays ({\tt copy} does not implement communication; if
it is passed non-aligned arrays, an exception occurs).

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs2 p = new Procs2(P, P) ;
  on(p) {
    Range x = new ExtBlockRange(N, p.dim(0), 1, 1) ;
    Range y = new ExtBlockRange(N, p.dim(1), 1, 1) ;

    float [[-,-]] a = new float [[x, y]] ;

    ...
    
    // Main loop.

    float [[-,-]] b = new float [[x, y]], r = new float [[x, y]] ;

    do {
      Adlib.writeHalo(a) ;

      overall(i = x for 1 : N - 2)
        overall(j = y for 1 : N - 2) {
          float newA ;

          newA = 0.25 * (a [i - 1, j] + a [i + 1, j] +
                         a [i, j - 1] + a [i, j + 1]) ;

          r [i, j] = Math.abs(newA - a [i, j]) ;
          b [i, j] = newA ;
        }

      HPspmd.copy(a, b) ;

    } while(Adlib.maxval(r) > EPS) ;

    ...
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Solution of Laplace equation using ghost regions.%
\label{fig:ghostJacobi}}
\end{figure*}


As a matter of fact, if we removed the temporary, {\tt b},
and reassigned the {\tt a} elements on-the-fly in terms of the
other partially updated {\tt a} elements, nothing very bad would happen.
The algorithm may even converge faster because it is locally using the
more efficient {\em Gauss-Siedel} relaxation scheme.
Figure \ref{fig:redBlack} is an implementation of the ``red-black''
scheme (a true Gauss-Siedel scheme), in which all even sites are
updated, then all odd sites are updated in a separate phase.  There is
no need to introduce the temporary array {\tt b}.  This example
illustrates the use of a stepped triplet in an {\tt overall} construct.

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
    do {
      for(int parity = 0 ; parity < 2 ; parity++) {

        Adlib.writeHalo(u) ;

        overall(i = x for 1 : N - 2)
          overall(j = y for 1 + (i` + parity) % 2 : N - 2 : 2) {
            float newA ;

            newA = 0.25 * (a [i - 1, j] + a [i + 1, j] +
                           a [i, j - 1] + a [i, j + 1]) ;

            r [i, j] = Math.abs(newA - a [i, j]) ;
            a [i, j] = newA ;
          }
      }

    } while(Adlib.maxval(r) > EPS)) ;
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Solution of Laplace equation using red-black relaxation
(main loop only).\label{fig:redBlack}}
\end{figure*}

The ``footprint'' of the stencil update can be more general.  Figure
\ref{fig:life} shows an implementation of Conway's Game of Life.  The
local update involves dependences on diagonal neighbours.  This example
also shows the most general form of {\tt writeHalo} library function,
which allows one to specify exactly how much of the available ghost
areas are to be updated (this can be less than the total ghost
area allocated for the array) and to specify a ``mode'' of updating the ghost
cells at the extremes of the whole array.  By specifying {\tt CYCLIC}
mode, cyclic boundary conditions are automatically implemented.

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
int wlo [] = {1, 1}, whi [] = {1, 1} ;  // ghost widths for `writeHalo'
int mode [] = {CYCL, CYCL} ;            // boundary conds for   "

Procs2 p = new Procs2(2, 2) ;
on(p) {
  Range x = new ExtBlockRange(N, p.dims(0), 1, 1) ;
  Range y = new ExtBlockRange(N, p.dims(1), 1, 1) ;

  int [[-,-]] state = new int [[x, y]] ;

  ... Define initial state of Life board

  // Main update loop.

  int sums [[-,-]] = new int [[x, y]] ;

  for(int iter = 0 ; iter < NITER ; iter++) {

    Adlib.writeHalo(state, wlo, whi, mode) ;

    // Calculate neighbour sums.

    overall(i = x for :)
      overall(j = y for :)
        sums [i, j] =
          state [i - 1, j - 1] + state [i - 1, j] + state [i - 1, j + 1] +
          state [i,     j - 1]                    + state [i,     j + 1] +
          state [i + 1, j - 1] + state [i + 1, j] + state [i + 1, j + 1] ;

    // Update state of board values.

    overall(i = x for :)
      overall(j = y for :)
        switch (sums [i, j]) {
          case 2 : break;
          case 3 : state [i, j] = 1; break;
          default: state [i, j] = 0; break;
        }
  }

  ... Output final state
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Conway's Game of Life.\label{fig:life}}
\end{figure*}

As a final example, Figure \ref{fig:ising} is a Monte Carlo simulation
of the well-known Ising model from condensed matter physics.  It combines
cyclic boundary conditions with red-black updating scheme.  Random numbers
are generated here using the {\tt Random} class from the standard Java
library.  Random streams are created with different values in each process
using some expression that depends on the local coordinate value. 
(The method used here is certainly too naive for a reliable simulation,
but it illustrates the principle.)

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  int wlo [] = {1, 1}, whi [] = {1, 1} ;
  int mode [] = {CYCL, CYCL} ;

  Procs2 p = new Procs2(P, P) ;
  on(p) {
    Range x = new ExtBlockRange(N, p.dim(0), 1, 1) ;
    Range y = new ExtBlockRange(N, p.dim(1), 1, 1) ;

    float [[-,-]] latt = new float [[x, y]] ;

    Random rand = new Random(97    * p.dim(0).crd() +
                             89793 * p.dim(1).crd()) ;

    ... Initialize `latt' randomly with +1's and -1's.

    // Main loop.

    for (int sweep = 0 ; sweep < NSWEEPS ; sweep++) {

      for(int parity = 0 ; parity < 2 ; parity++) {

        Adlib.writeHalo(latt, wlo, whi, mode) ;

        overall(i = x for :)
          overall(j = y for (i` + parity) % 2 : : 2) {

            int oldVal = latt [i, j] ;
            int newVal = rand.nextFloat() < 0.5 ? -1 : 1 ;
                                      // Randomly choose +1 or -1

            int deltaE = (newVal - oldVal) *
                             (latt [i - 1, j] + latt [i + 1, j] +
                              latt [i, j - 1] + latt [i, j + 1]) ;

            if(rand.nextFloat() < Math.exp(- BETA * deltaE))
              latt [i, j] = newVal ;    // Accept, biased by energy
          }
      }
    }

    ... Analyse final configuration
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Monte Carlo simulation of Ising model using the Metropolis algorithm.%
\label{fig:ising}}
\end{figure*}

\clearpage

\section{Collapsed Distributions and Sequential Dimensions\label{sec:collapsed}}

The {\tt CollapsedRange} subclass in Figure \ref{fig:ranges} stands for
a range that is {\em not} distributed---all elements of the range
are mapped to a single process.  The code
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs1 q = new Procs1(3) ;
  on(p) {
    Range x = new CollapsedRange(N) ;
    Range y = new BlockRange(N, q.dim(0)) ;

    float [[-,-]] a = new float [[x, y]] ;

    ...
  }
\end{verbatim}
\end{code}
\end{displaymath}
creates an array in which the second dimension is distributed over
processes in {\tt q}, with the first dimension {\em collapsed}.  The
situation is visualized for the case {\tt N} = 8
in Figure \ref{fig:array1}.  (This is our first example of a 
one-dimensional process ``grid''.)

\begin{figure*}[tbp]
%\centerline{\psfig{figure=array1.eps,width=4in}}
\centerline{\includegraphics[width=4in]{array1.eps}}
\caption{Two-dimensional array, {\tt a}, distributed over the one-dimensional
grid, {\tt q}.\label{fig:array1}}
\end{figure*}

%Collapsed dimensions are often useful in algorithms where the pattern
%of access to elements is irregular in some but not all array dimensions.
%In the example above a process can access any element along
%the {\tt x} dimension without communication, providing a fixed {\tt y}
%location is mapped to the process concerned.

Unfortunately the language defined so far doesn't provide a good way
to exploit the locality implied by a collapsed dimension.  
If we want to assign the value in {\tt a[6,1]}
to {\tt a[2,1]} we have to do something convoluted like
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs1 q = new Procs1(3) ;
  on(p) {
    Range x = new CollapsedRange(N) ;
    Range y = new BlockRange(N, q.dim(0)) ;

    float [[-,-]] a = new float [[x, y]] ;
    ...
    at(j = y [1]) {
      float local ;

      at(i = x [6])
        local = a [i, j] ;

      at(i = x [2])
        a [i, j] = local ;
    }
  }
\end{verbatim}
\end{code}
\end{displaymath}
Because of the restriction that subscripts must be distributed indices, the
value of {\tt a[6,1]} must first be read to a local variable in an
{\tt at} construct, then the value of the local variable must be copied to
{\tt a[2,1]} in another.  This is very tedious, and probably
inefficient.

To avoid this common problem, the HPJava model of distributed arrays
is extended with the idea of {\em sequential dimensions}.  If the type
signature of a distributed array has an asterix in a particular dimension,
that dimension will implicitly have a collapsed range, and the dimension
can be subscripted with integer expressions just like a sequential array.
The example becomes
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs1 q = new Procs1(3) ;
  on(p) {
    Range y = new BlockRange(N, q.dim(0)) ;

    float [[*,-]] a = new float [[N, y]] ;
    ...
    at(j = y [1])
      a [6, j] = a [1, j] ;
  }
\end{verbatim}
\end{code}
\end{displaymath}
The outer {\tt at} construct is retained to deal with the distributed dimension,
but there is no need for distributed indices in the sequential dimension.
The array constructor is passed integer extent expressions for
sequential dimensions.  A {\tt CollapsedRange} object will be created
for the array, but the programmer generally need not be aware of its
existence.

\begin{figure}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs1 p = new Procs1(P) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;

    float [[-,*]] a = new float [[x, N]], c = new float [[x, N]] ;
    float [[*,-]] b = new float [[N, x]], tmp = new float [[N, x]] ;

    ... initialize `a', `b'

    for(int s = 0 ; s < N ; s++) {

      overall(i = x for :) {

        float sum = 0 ;
        for(int j = 0 ; j < N ; j++)
          sum += a [i, j] * b [j, i] ;

        c [i, (i` + s) % N] = sum ;
      }

      // cyclically shift `b' (by amount -1 in x dim)...

      Adlib.cshift(tmp, b, -1, 0) ;
      HPspmd.copy(b, tmp) ;
    }
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{A pipelined matrix multiplication program.\label{fig:pipedMatmul}}
\end{figure}

Figure \ref{fig:pipedMatmul} is an example of a parallel matrix
multiplication in which the first input array, {\tt a}, and result
array, {\tt c}, are distributed by rows---each processor is allocated a
consecutive set of complete rows.  The first dimension of these array
is distributed, breaking up the columns, while the second dimension is
collapsed, leaving individual rows intact.  The second input array,
{\tt b}, is distributed by columns.

Array types with sequential dimensions are technically subtypes of
corresponding array types without sequential dimension.  All operations
generally applicable to distributed arrays are also applicable to
arrays with sequential dimensions.  The asterisk in the type signature
adds the option of subscripting the associated with integer
espressions.  It does not remove any option allowed for distributed
arrays in general.

%This appears to be clearly the best way to define the language, but
%it introduces is a small pitfall for the unwary.  Consider this code
%\begin{displaymath}
%\begin{code}
%\begin{verbatim}
%    int a [[-,*]] = new int [[x, N]] ;
%
%    overall(i = x for :) {
%      a [i, i`] = 1 ;
%    }
%\end{verbatim}
%\end{code}
%\end{displaymath}
%This is perfectly good code, which initializes all elements on the
%diagonal of the distributed array, {\tt a}, with the value 1.  The
%pitfall is that one might write, mistakenly
%\begin{displaymath}
%\begin{code}
%\begin{verbatim}
%    overall(i = x for :) {
%      a [i, i] = 1 ;
%    }
%\end{verbatim}
%\end{code}
%\end{displaymath}
%{\tt a} is a distributed array, and as for any distributed array it is
%{\em syntactically} allowed to use a location for the second subscript,
%notwithstanding that the dimension is actually sequential.  The
%second fragment is illegal and would certainly lead to a runtime
%exception, because the location {\tt i} is not a location in the
%collapsed second range of the array.  But this problem may
%not be flagged at compile-time, because it is not in the nature
%of a ``type error''%
%\footnote{This kind of runtime exception is not peculiar to
%collapsed dimensions---it will occur in fully distributed cases if, for
%example, we mistakenly transpose a pair of distributed array
%subscripts.  The only extra danger with a sequential dimension is that
%omission of a single backquote symbol is a little too easy, and looks
%a little too innocuous.}%
%.

\section{Replication and Distribution Groups\label{sec:replicated}}

Allowing collapsed array dimensions means that an array can be
distributed over a process grid having smaller rank than the array
itself.  Conversely it is also allowed to distribute an array over a
process grid whose rank is larger than the array.  
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs2 p = new Procs2(P, P) ;

  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;

    float [[-]] b = new float [[x]] ;

    ...
  }
\end{verbatim}
\end{code}
\end{displaymath}
The array {\tt b} has a dimension distributed over the first dimension
of {\tt p}, but none distributed over the second.  The interpretation
is that {\tt b} is {\em replicated} over the second process dimension.
Independent copies of the whole array are created at each coordinate
where replication occurs.  Usually programs maintain identical
values for the elements in each copy (although there is nothing in the
language definition itself to require this).

Replication and collapsing can both occur in a single array, for example
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs2 p = new Procs2(P, P) ;

  on(p) {
    float [[*]] c = new float [[N]] ;

    ...
  }
\end{verbatim}
\end{code}
\end{displaymath}
The range of {\tt c} is sequential, and the array is replicated over
both dimensions of {\tt p}.  This makes it very similar to an ordinary
Java array declared in all processes by
\begin{displaymath}
\begin{code}
\begin{verbatim}
    float [] d = new float [N] ;
\end{verbatim}
\end{code}
\end{displaymath}
The properties of {\tt c} and {\tt d} are not identical, though.
The array {\tt c} can be passed to library functions that expect
distributed arrays as arguments, whereas {\tt d} cannot.

In the last section we saw a ``pipelined'' matrix multiply algorithm.
A simpler and potentially more efficient implementation of matrix
multiplication can be given if the operand arrays have carefully
chosen replicated/collapsed distributions.  The program is given
in Figure \ref{fig:directMatmul}.  As illustrated in Figure
\ref{fig:directMatDist}, the rows of {\tt a} are replicated in
the process dimension associated with {\tt y}.  Similarly the
columns of {\tt b} are replicated in the dimension associated with
{\tt x}.  Hence all arguments for the inner scalar product are
already in place for the computation---no communication is needed.

\begin{figure}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs2 p = new Procs2(P, P) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;
    Range y = new BlockRange(N, p.dim(1)) ;

    float [[-,-]]  c = new float [[x, y]] ;

    float [[-,*]] a = new float [[x, N]] ;
    float [[*,-]] b = new float [[N, y]] ;

    ... initialize `a', `b'

    overall(i = x for :)
      overall(j = y for :) {

        float sum = 0 ;
        for(int k = 0 ; k < N ; k++)
          sum += a [i, k] * b [k, j] ;

        c [i, j] = sum ;
      }
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{A direct matrix multiplication program.\label{fig:directMatmul}}
\end{figure}

\begin{figure*}[tbp]
%\centerline{\psfig{figure=matDist.eps,width=3.5in}}
\centerline{\includegraphics[width=3.5in]{matDist.eps}}
\caption{Distribution of array elements in example of Figure
\ref{fig:directMatmul}.  Array {\tt a} is replicated in every column
of processes, array {\tt b} is replicated in every row.}
\label{fig:directMatDist}
\end{figure*}

We would be very lucky to come across three arrays with such a
special {\em alignment relation}
(distribution format relative to one another).
There is an important function in the Adlib library called {\tt remap},
which takes a pair of arrays as arguments.  These must have the same
shape and type, but they can have unrelated distribution formats.  The
elements of the source array are copied to the destination array.  In
particular, if the destination array has a replicated mapping, the
values in the source array are broadcast appropriately.

\begin{figure}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  void matmul(float [[-,-]] c, float [[-,-]] a, float [[-,-]] b) {

    Group p = c.grp() ;

    Range x = c.rng(0) ;
    Range y = c.rng(1) ;

    int N = a.rng(1).size() ; 

    float [[-,*]] ta = new float [[x, N]] on p ;
    float [[*,-]] tb = new float [[N, y]] on p ;

    Adlib.remap(ta, a) ;
    Adlib.remap(tb, b) ;

    on(p)
      overall(i = x for :)
        overall(j = y for :) {

          float sum = 0 ;
          for(int k = 0 ; k < N ; k++)
            sum += ta [i, k] * tb [k, j] ;

          c [i, j] = sum ;
        }
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{A general matrix multiplication program.\label{fig:generalMatmul}}
\end{figure}

Figure \ref{fig:generalMatmul} shows how we can use {\tt remap}
to adapt the program in Figure \ref{fig:directMatmul} and create
a general purpose matrix multiplication routine.  Besides the
{\tt remap} function, this example introduces the two inquiry
methods {\tt grp()} and {\tt rng()} which are defined for any distributed
array.  The inquiry {\tt grp()} returns the distribution group of the
array, and the inquiry $\mbox{\tt rng(}r\mbox{\tt )}$ returns the $r$th
range of the array.  The argument $r$ is in the range $0, \ldots, R - 1$,
where $R$ is the rank (dimensionality) of the array.

The example also illustrates the most general form of the distributed
array constructor.  In all earlier examples arrays were distributed
over the whole of the active process group, defined by an enclosing {\tt on}
construct.  In general an {\em on clause} attached to an array constructor
itself can specify that the array is distributed over some subset of the
active group.  This allows one to create an array outside the
{\tt on} construct that will processes its elements.  Through communication
functions like {\tt remap}, values can then be exchanged between different
process grids.  A sketch example is given in Figure \ref{fig:gridExchange}.


\begin{figure}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs2 p = new Procs2(P, P) ;

  Range x = new BlockRange(N, p.dim(0)),
        y = new BlockRange(N, p.dim(1)) ;

  float [[-,-]] a = new float [[x, y]] on p;

  on(p) {
    ... compute elements in 2d block-distributed `a' ...
  }


  Procs1 q = new Procs1(Q) ;

  Range z = new BlockRange(N, q.dim(0));

  float [[-,*]] b = new float [[z, N]] on q;


  Adlib.remap(b, a);  // copy elements of `a' to row-distributed `b'

  on(q) {
    ... process elements of `b' ...
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Sketch example exchanging data between different
         grids.\label{fig:gridExchange}}
\end{figure}

To allow for this kind of situation, where arguments might be distributed
over distinct process groups---not the active process group---the
generic matrix multiplication of Figure \ref{fig:generalMatmul} included
{\tt on p} clauses in the constructors of its temporary arrays,
and explicitly restricts control with an {\tt on(p)} construct before
processing.  As we will see in the next section, this refinement also
allows the arguments of {\tt matmul} to be arbitrary array {\em sections}.


\chapter{Array Sections\label{sec:sections}}

HPJava has a mechanism for representing subarrays.  This mechanism is
modelled on the {\em array sections} of Fortran 90.  In HPJava an array
section expression has a similar syntax to a distributed array element
reference but uses double brackets.  Whereas an element
reference is a variable, an array section is an
expression that represents a new distributed array object.  

The new array does not contain new elements.  It contains a subset of the elements
of the parent array.  Those
elements can subsequently be referenced---read or updated---either
through the parent array {\em or} through the array section%
\footnote{The HPJava idea of an array section expression has a close
relationship to the Fortran 90 idea of an array pointer.  In Fortran an
array pointer can reference an arbitrary regular section of an array}%
.

We have seen that the subscripts in a distributed array element reference are
either locations or (restrictedly) integer expressions.  Options for
subscripts in array section expressions are wider.  Most importantly,
as in Fortran 90, a section subscript is allowed be a triplet.
For each triplet subscript a section
expression has an array dimension.  So in a normal array
section expression the rank of the result is equal to the
number of triplet subscripts.  The section may also have some scalar
subscripts, similar to those appearing in element references.  In this
case the rank of the result will be lower than the rank of the parent
array.

This fragment includes two examples of array section expressions:
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs2 p = new Procs2(P, P) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;
    Range y = new BlockRange(N, p.dim(1)) ;

    float [[-,-]] a = new float [[x, y]] ;

    float [[-]] b = a [[0, :]]

    foo(a [[0 : N / 2 - 1,  0 : N - 1 : 2]]) ;
  }
\end{verbatim}
\end{code}
\end{displaymath}
The first array section expression appears on the right hand side of
the definition of {\tt b}.  It specifies {\tt b} as an alias for the
first row of {\tt a} (Figure \ref{fig:section1}).
In an array section expression, {\em unlike} in an array element reference, 
a scalar subscript is always allowed to be an integer expression%
\footnote{Distributed indices are allowed as well, if the location
is in the proper range.}%
.  The second array
section expression appears as an argument to the method {\tt foo}.  It
represents a two-dimensional, $N/2$ by $N/2$, subset of the
elements of {\tt a}, visualized in Figure \ref{fig:section2}.

\begin{figure*}[tbp]
%\centerline{\psfig{figure=section1.eps,width=4in}}
\centerline{\includegraphics[width=4in]{section1.eps}}
\caption{A one-dimensional section of a two-dimensional array
(shaded area).\label{fig:section1}}
\end{figure*}

\begin{figure*}[tbp]
%\centerline{\psfig{figure=section2.eps,width=4in}}
\centerline{\includegraphics[width=4in]{section2.eps}}
\caption{A two-dimensional section of a two-dimensional array
(shaded area).\label{fig:section2}}
\end{figure*}

Array sections allow us to implement a number of interesting
applications.  They are often passed as arguments to library functions
like {\tt remap}, implementing various interesting patterns of
communication and arithmetic on subarrays.

\section{Two-dimensional Fourier transform}

In image processing applications Fast Fourier Transforms (FFTs) and
related transformations are sometimes applied to two-dimensional
images.  A two-dimensional FFT can be broken down into a series simpler
one-dimensional FFTs---the one-dimensional transform is simply applied
to every row of the image, then to every column.  All rows can
be transformed in parallel, then all columns can be transformed in
parallel.  An implementation is sketched in Figure \ref{fig:fft2d}
This implementation assumes the availability of a function for
calculating FFTs on one-dimensional {\em collapsed} arrays (ie,
a sequential one-dimensional FFT.  Because Java doesn't have complex
numbers, we store real and imaginary parts in pairs of arrays whose
names are prefixed {\tt re} and {\tt im}).  Alternatively a
an extra dimension of extent 2 could be added to the arrays.
After processing all columns, the data is remapped so that each row is
a collapsed subarray.  A section dimension naturally inherits the
sequential property (the asterisk in the type signature) from the
associated dimension of the parent array.

\begin{figure}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  void fft1d(float [[*]] re, float [[*]] im) {

    // One-dimensional FFT on sequential (non-distributed) data.

    ...
  }

  Procs1 p = new Procs1(P) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;

    float [[-,*]] reA = new float [[x, N]], imA = new float [[x, N]] ;
    float [[*,-]] reB = new float [[N, x]], imB = new float [[N, x]] ;

    ... initial values in `reA', `imA'

    overall(i = x for :)
      fft1d(reA [[i, :]], imA [[i, :]]) ;

    Adlib.remap(reB, reA) ;
    Adlib.remap(imB, imA) ;

    overall(i = x for :)
      fft1d(reB [[:, i]], imB [[:, i]]) ;

    ... result is in `reB', `imB'
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{A two-dimensional Fourier Transform.\label{fig:fft2d}}
\end{figure}

\section{Cholesky decomposition}

If $A$ is a symmetric positive definite matrix, associated
linear equations are often solved using Choleski decomposition:
\begin{displaymath}
A=LL{^T}
\end{displaymath}
where $L$ is a lower triangular matrix.  In practise this is
followed by forward and back substitutions:
\begin{displaymath}
\mathit{L}\mathbf{y}=\mathbf{b}, \; \mathit{L{^T}}\mathbf{x}=\mathbf{y}
\end{displaymath}
to complete the solution of $A\mathbf{x}=b$.
A pseudocode algorithm for Cholesky decomposition is
\begin{displaymath}
\begin{array}{l}
For\; k=1\; to\; n-1 \\
\;\;\;\; l_{kk}=a_{kk}^{1/2} \\
\;\;\;\; For\; s=k+1\; to\; n \\
\;\;\;\;\;\;\;\; l_{sk}=a_{sk}/l_{kk} \\
\;\;\;\; For\; j=k+1\; to\; n \\
\;\;\;\;\;\;\;\; For\; i=j\; to\; n \\
\;\;\;\;\;\;\;\;\;\;\;\; a_{ij}=a_{ij}-l_{ik}l_{jk} \\
l_{nn}=a_{nn}^{1/2} \\
\end{array}
\end{displaymath}
A parallel version, assuming the main array is stored by columns
with the rows cyclically distributed, is given in figure \ref{fig:cholesky}.
The $l$ array is accumulated in the lower part of the input array {\tt
a}.  Note that the array {\tt b} has a replicated distribution, so the
{\tt remap} operation is a broadcast of the relevant part of column
$k$.

{\em  [Discuss the expression {\tt b[j`]}---why it must be {\tt j`}.]}

\begin{figure}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs1 p = new Procs1(P) ;
  on(p) {
    Range x = new CyclicRange(N, p.dim(0));

    float [[*,-]] a = new float [[N, x]] ;

    float [[*]]   b = new float [[N]] ;  // a buffer

    ... some code to initialise `a'

    for(int k = 0 ; k < N - 1 ; k++) {

      at(j = x [k]) {
        float d =  Math.sqrt(a [k, j]) ;

        a [k, j] = d ;
        for(int s = k + 1 ; s < N ; s++)
          a [s, j] /= d ;
      }

      Adlib.remap(b [[k + 1 : ]], a [[k + 1 : , k]]);

      overall(j = x for k + 1 : )
        for(int i = j` ; i < N ; i++)
          a [i, j] -= b [i] * b [j`] ;
    }

    at(j = x [N - 1])
      a [N - 1, j] = Math.sqrt(a [N - 1, j]) ;
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Cholesky decomposition.\label{fig:cholesky}}
\end{figure}

\section{Matrix multiplication with reduced memory}

One disadvantage of the program in Figure \ref{fig:generalMatmul}
is that it allocates two very large temporary arrays, {\tt ta} and {\tt
tb}.  Because these are both replicated in one dimension, they can
easily consume more memory than the original arguments.  This problem can be
solved by only storing copies of elements in restricted bands of the
original matrices at any one time.  Figure \ref{fig:reducedMatmul}
gives a modified algorithm where the maximum band width is {\tt B}.

\begin{figure}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  void matmul(float [[-,-]] c, float [[-,-]] a, float [[-,-]] b) {

    Group p = c.grp() ;

    Range x = c.rng(0) ;
    Range y = c.rng(1) ;

    int N = a.rng(1).size() ;

    float [[-,*]] ta = new float [[x, B]] on p ;
    float [[*,-]] tb = new float [[B, y]] on p ;

    on(p)
      overall(i = x for :)
        overall(j = y for :)
          c [i, j] = 0 ;

    for(int base = 0 ; base < N ; base += B) {
      int w = min(B, N - base) ;  // minimum value

      Adlib.remap(ta [[:, 0 : w - 1]], a [[:, base : base + w - 1]]) ;
      Adlib.remap(tb [[0 : w - 1, :]], b [[base : base + w - 1, :]]) ;

      on(p)
        overall(i = x for :)
          overall(j = y for :)
            for(int k = 0 ; k < w ; k++)
              c [i, j] += ta [i, k] * tb [k, j] ;
    }
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Matrix multiplication with reduced memory requirement.%
\label{fig:reducedMatmul}}
\end{figure}

For simplicity we assumed here that
{\tt B} is a compile-time constant.  Alternatively we can compute
this value dynamically.
The {\tt volume()} method on {\tt Range} is used internally by array
constructors to control allocation of memory for array elements.  It defines
the largest block of locations of the current range held by any processor. 
Hence an upper bound on the number of elements held by any processor
for {\tt ta} and {\tt tb} combined is 
\begin{displaymath}
\begin{code}
\begin{verbatim}
    B * x.volume() + B * y.volume()
\end{verbatim}
\end{code}
\end{displaymath}
If {\tt MAX\_TEMPORARY\_SIZE} is a constant defining a limit on the
total volume of memory we ever wish to allocate for temporary arrays,
a suitable formula for {\tt B} might be
\begin{displaymath}
\begin{code}
\begin{verbatim}
    B = MAX_TEMPORARY_SIZE / (x.volume() + y.volume())
\end{verbatim}
\end{code}
\end{displaymath}
With a few refinements like this, the algorithm of Figure
\ref{fig:reducedMatmul} becomes a credible basis for a library
matrix multiplication routine, applicable to generic distributed
arrays.

\section{Subranges}

Consider again the example of the array section in figure
\ref{fig:section2}.  We can capture this object
in a named variable as follows
\begin{displaymath}
\begin{code}
\begin{verbatim}
    float [[-,-]] a = new float [[x, y]] ;

    float [[-,-]] c = a [[0 : N / 2 - 1, 0 : N - 1 : 2]] ;
\end{verbatim}
\end{code}
\end{displaymath}
Now, what are the ranges of {\tt c}---the objects returned
by the {\tt rng()} inquiry applied to {\tt c}?

In fact they are a different sort of range from any considered so
far---they are {\em subranges}.  For completeness the HPJava language
provides a special syntax for constructing subranges directly.
Ranges equivalent to those of {\tt c} can be created by
\begin{displaymath}
\begin{code}
\begin{verbatim}
    Range u = x [0 : N / 2 - 1] ;
    Range v = y [0 : N - 1 : 2] ;
\end{verbatim}
\end{code}
\end{displaymath}
This syntax should look quite natural.  It is similar to the subscripting
syntax for locations, but the subscript is a triplet.

The global indices associated with the subrange $v$, for example,
are in the range
$0, \ldots, v \mbox{\tt .size()}$.  A subrange inherits locations from
its parent range, but it specifically {\em does not} inherit global
indices from the parent.

A {\em non-trivial subrange} is one for which the lower bound is not
equal to zero, or the upper bound is not equal to ${\tt size()} - 1$,
or the stride is not equal to 1.  

A non-trivial subrange is never considered to have ghost extensions,
even if its parent range does.  This avoids various ambiguities that
might otherwise crop up.

What about the distribution groups of sections?  Now
triplet subscripts don't cause problems---the distribution group
of {\tt c} above can be defined to be the same as the distribution
group of the parent array {\tt a}.  But the example of figure
\ref{fig:section1} is problematic.  This was
constructed using a scalar subscript, effectively as follows:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    float [[-,-]] a = new float [[x, y]] on p ;

    float [[-]] b = a [[0, :]]
\end{verbatim}
\end{code}
\end{displaymath}
The single range of {\tt b} is clearly {\tt y}, but identifying the
distribution group of {\tt b} with that of {\tt a} doesn't seem to be right.
If a one dimensional array is newly constructed with range {\tt y} and
distribution group {\tt p}, like this:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    float [[-]] bnew = new float [[y]] on p ;
\end{verbatim}
\end{code}
\end{displaymath}
it is understood to be replicated over the first dimension of {\tt p}.
The section {\tt b} clearly isn't replicated in this way.  Where does
the information that {\tt b} is localized to the top row of processes go?

\section{Restricted Groups}

In the last section triplet section subscripts motivated us to define
subranges as a new
kind of range.  Likewise, scalar section subscripts will drive us to define
a new kind of group.  A {\em restricted group} is defined to be
the subset of processes in some parent group to which a particular
location is mapped.  In the current example, the distribution group of
{\tt b} is defined to be the subset of processes in {\tt p} to which
the location {\tt x[0]} is mapped.  Rather than of further extending
subscripting notations to describe these subgroups, the division operator is
overloaded.  The distribution group of {\tt b} is equivalent to {\tt
q}, defined by
\begin{displaymath}
\begin{code}
\begin{verbatim}
    Group q = p / x [0] ;
\end{verbatim}
\end{code}
\end{displaymath}
The expression in the initializer is called a {\em group restriction}
operation%
%\footnote{Choice of an arithmetic operator rather than a subscripting
%notation for group restriction is not ad hoc.  A subscripting notation
%would require (compile-time) information about the position of a
%particular process dimension in the list of dimensions of the group.
%Experience shows that this positional information is often not fixed
%``statically''---the process dimension often varies from invocation to
%invocation of a given piece of code.}%
.

In a sense the definition of a restricted group is tacit in the definition
of an abstract location.  Without formally defining the idea, we
used it implicitly in section \ref{sec:locations}.  In Figure
\ref{fig:locations} of that section the set of processes with
coordinates $(0,0)$, $(0,1)$ and $(0,2)$, to which location {\tt x[1]}
is mapped, can now be written as
\begin{displaymath}
\begin{code}
\begin{verbatim}
    p / x [1]
\end{verbatim}
\end{code}
\end{displaymath}
and the set
with coordinates $(0,1)$ and $(1,1)$, to which {\tt y[4]} is mapped,
can be written as
\begin{displaymath}
\begin{code}
\begin{verbatim}
    p / y [4]
\end{verbatim}
\end{code}
\end{displaymath}
The intersection of these two---the group containing the single
process with coordinates $(0,1)$---can be written as
\begin{displaymath}
\begin{code}
\begin{verbatim}
    p / x [1] / y [4]
\end{verbatim}
\end{code}
\end{displaymath}
or as
\begin{displaymath}
\begin{code}
\begin{verbatim}
    p / y [4] / x [1]
\end{verbatim}
\end{code}
\end{displaymath}

At first sight the definition of HPJava restricted groups may appear slightly
arbitrary.  One good way to argue that a language construct is
``natural'' is to demonstrate that it has a simple and efficient
implementation.  The subgroups introduced here have an attractively
simple concrete representation.  A restricted group is uniquely specified by
its set of effective process dimensions and the identity of the {\em
lead} process in the group---the process with coordinate zero relative
to the dimensions effective in the group.  The dimension set can be
specified as a subset of the dimensions of the parent grid using a
simple bitmask.  The identity of the lead process can be specified
through a single integer ranking the processes of the parent grid.  So
a general HPJava group can be parametrized by a reference to the parent
{\tt Procs} object, plus just two {\tt int} fields.  It turns out that
this representation is not only compact; it also lends itself to efficient
computation of the most commonly used operations on groups.

Note that the inquiry function {\tt dim()} is a member of the {\em Procs}
class (the process grid class), {\em not} the superclass {\tt Group}.
%An inquiry {\tt prc()} on {\tt Group} returns the parent process grid of
%the group; if essential the process dimensions associated with a general
%group can be retrieved through the grid object returned by {\tt prc()}.

\section{Mapping of array sections}

Now we can give a formal definition of the mapping (distribution
group and ranges) of a general array section.  

If the $r$th dimension of array $a$ is non-sequential, an integer
section subscript, $n$, in this dimension behaves like a location-valued
subscript, $a \mbox{\tt .rng(} r \mbox{\tt )[} n \mbox{\tt ]}$%
\footnote{If the $r$th dimension is sequential, this equivalence is
not strict; an integer subscript in a sequential dimension may
have extended bounds in the peculiar case that the sequential
dimension has ghost extensions.  But there are no locations associated
with subscript values in ghost extensions.}%
.  Suppose
any such integer subscripts are replaced by their equivalent location
subscripts in this way.  If the set of all location subscripts is now $i,
j, \ldots$, the distribution group of the section is
\begin{displaymath}
    p / i / j / \ldots
\end{displaymath}
where $p$ is the distribution group of the parent array.
For a shifted index, as a matter of definition,
\begin{displaymath}
\begin{code}
\begin{tabbing}
\verb$    p / (i $$\pm$\verb$ ${\em expression}\verb$)  =  p / i$
\end{tabbing}
\end{code}
\end{displaymath}
This makes sense---a shifted index is supposed to find an array
element in the same process as the original location, albeit that
the element could be in a ghost region.

The $k$th range of the section is determined by the $k$th triplet-valued
subscript.
If the $k$th triplet-valued subscript is 
$l \mbox{\tt :} u \mbox{\tt :} s$ in dimension $r$, the
$k$th range of the section is 
$a \mbox{\tt .rng(} r \mbox{\tt )[} l \mbox{\tt :} u \mbox{\tt :} s \mbox{\tt ]}$.

Note that, because non-trivial subranges are never considered to have ghost
extensions, a section constructed with non-trivial triplet subscripts in
some dimensions is not be considered to have accessible ghost extensions
in those dimensions, even if its parent array had them.

It shouldn't come as a surprise that subranges and restricted groups can be
used in array constructors, on the same footing as the ranges and
groups described in earlier sections.  This means, for example, that
temporary arrays can be constructed with identical mapping to any
given section.  This facility is useful when writing generic library
functions, such as the {\tt matmul} of Figure \ref{fig:generalMatmul},
which must accept full arrays or array sections without discrimination%
\footnote{It also allows HPJava arrays to reproduce the full
panoply of alignment options supported by the {\tt ALIGN} directive of
High Performance Fortran.}%
.

The last three sections were unusual in that they introduced some new
pieces of syntax but did not give any full example programs that use them.
The reason is that restricted groups and subranges largely exist ``below
the surface'' in HPJava.  The new notations are mainly needed to add
a kind of semantic completeness to the language.  It is not especially
common to see subgroups or subranges constructed explicitly in HPJava
programs.

\section{Scalars}

We imposed no restriction that the list of subscripts in an array
section expression {\em must} include some triplets (or ranges).  It is
legitimate for all the subscript to be ``scalar''.  In this case the
resulting ``array'' has rank 0.

There is nothing pathological about rank-0 arrays.  They logically
maintain a single element, bundled with the distribution group over which
this element is replicated.  Because they are logically distributed
arrays they can be passed as arguments to Adlib functions such as
{\tt remap}.  If {\tt a} and {\tt b} are distributed arrays, we
cannot usually write a statement like
\begin{displaymath}
\begin{code}
\begin{verbatim}
    a [10, 10] = b [30] ;
\end{verbatim}
\end{code}
\end{displaymath}
because the elements involved are generally held on different processors.
As we have seen, HPJava imposes constraints that forbid this kind of  
direct assignment between array element references.  However, if it is really
needed, we can usually achieve the same effect by writing
\begin{displaymath}
\begin{code}
\begin{verbatim}
    Adlib.remap(a [[10, 10]], b [[30]]);
\end{verbatim}
\end{code}
\end{displaymath}
The arguments are rank-0 sections holding just the destination and
source elements.

The story of subranges and restricted groups repeats itself.  The
operation of array sectioning drives us to introduce a new kind of
object into the language.  Once that happens we should have
a syntax for creating the new kind of object directly.
Rank-0 distributed arrays, which we will also call simply ``scalars'',
can be created as follows
\begin{displaymath}
\begin{code}
\begin{verbatim}
    float [[]] c = new float [[]] ;

    float a [[-,-]] = new float [[x, y]] ;

    Adlib.remap(c, a [[10, 10]]) ;

    float d = c [] ;
\end{verbatim}
\end{code}
\end{displaymath}
This example illustrates one way to broadcast an element of a
distributed array: remap it to a scalar replicated over the active
process group.  The element of the scalar is extracted through a
distributed array element reference with an empty subscript list.
Like any other distributed array, scalar constructors can have
{\tt on} clauses, specifying a non-default distribution group.

\section{Array restriction\label{sec:arrayRestriction}}

Library functions operating on distributed arrays often
specify certain {\em alignment relations} between their array
arguments.  In HPJava it is atural to define two arrays to
be {\em aligned} if they have the same distribution group and
all their ranges are aligned%
\footnote{Later we will give more detailed definitions.}%
.  
The Adlib member {\tt dotProduct}, for example, takes two distributed
array arguments.  These arguments must be aligned.

Occasionally it happens that two arrays we want to pass as arguments to
a library function are {\em essentially} aligned, but one is replicated
over a particular process dimension and the other isn't.  It may be
intuitively obvious that all the data needed by the function is in the
right place, but still we cannot call the function---the ranges may
match, but the replicated array has a larger distribution group.  By
the definition given above the arrays are not {\em identically}
aligned.

One possibility is to relax the definition of argument alignment
to take account of this situation.  
%Anyone writing a library
%callable from HPJava is free to take this path---they can write their
%functions to accept arrays with some weaker alignment constraint.
But experience suggests that the simple definition of alignment given
above is easy to understand, and the specification and implementation
of library functions are simpler if thay are based on this definition.

A minor extension to the HPJava language takes care of this situation.
The restriction operation introduced for groups in the previous section
can also be applied to an array.  It returns a new array
object---akin to an array section---which has the same ranges as the
parent array, but has its group restricted by the specified location.
Applied to a replicated array, it returns an array object referencing
only the copies of the elements held in the restricted group.

Figure \ref{fig:3dMatmul} is a generalization of the matrix multiplication
program in Figure \ref{fig:directMatmul} to the case where the arrays
are suitably distributed over a 3-dimensional process grid.  Note that
array {\tt c} is replicated over the process dimension of {\tt z}, {\tt
a} is replicated over the dimension of {\tt y}, and {\tt b} is
replicated over the dimension of {\tt x}.  The sequential inner loop of
Figure \ref{fig:directMatmul} is replaced by a call to {\tt dotProduct}
which directly forms the inner product of two sections with distributed
range {\tt z}.

\begin{figure}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs3 p = new Procs3(P, P, P) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;
    Range y = new BlockRange(N, p.dim(1)) ;
    Range z = new BlockRange(N, p.dim(2)) ;

    float [[-,-]] c = new float [[x, y]] ;

    float [[-,-]] a = new float [[x, z]] ;
    float [[-,-]] b = new float [[z, y]] ;

    ... initialize `a', `b'

    overall(i = x for :)
      overall(j = y for :)
        c [i, j] = Adlib.dotProduct(a [[i, :]] / j, b [[:, j]] / i) ;
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{A maximally parallel matrix multiplication program.%
\label{fig:3dMatmul}}
\end{figure}

If we didn't know about array restriction we would probably try
to write the loop body as
\begin{displaymath}
\begin{code}
\begin{verbatim}
        c [i, j] = Adlib.dotProduct(a [[i, :]], b [[:, j]]) ;
\end{verbatim}
\end{code}
\end{displaymath}
The trouble is that according to the rules of the previous section
the first argument of {\tt dotProduct} has distribution group
$\mbox{\tt p} / \mbox{\tt i}$ whereas the second has distribution group
$\mbox{\tt p} / \mbox{\tt j}$.  So the arrays
are not identically aligned.  By forming restricted versions of both
these sections we reduce both groups down to
$\mbox{\tt p} / \mbox{\tt i} / \mbox{\tt j}$.  Luckily
this is also the home group of the array element {\tt c [i, j]},
so the program will work correctly.

This is the first example we have given of a call to a collective
library function {\em inside} the parallel {\tt overall} construct.  The
library, Adlib, supports this kind of ``nested parallelism'' provided
a few precautions are taken.  These will be explained
in section \ref{sec:library}.


\chapter{Some Rules and Definitions\label{sec:rules}}

In the preceding chapter we completed the definition of the HPJava process
group by adding the idea of a {\em restricted group} to the earlier
idea of a process grid.  This development has some useful applications
to the basic distributed control constructs of the HPJava language.

\section{Rules for distributed control constructs}

In earlier sections we sometimes referred informally to the ``active
process group''.  One concrete role of this group was as the default
distribution target in distributed array constructors.  We used
the fact that the {\tt on} construct establishes its group argument as the
active process group inside the body of the construct.
The other distributed control constructs, {\tt at} and {\tt overall}, also
affect the active process group---recall the discussion
in section \ref{sec:locations}.  We can use the notations introduced
in the last section to state their effect more formally

First, for completeness, we restate the effect of the {\tt on} construct
and give an associated rule.  As explained in section
\ref{sec:processes}, the construct
\begin{displaymath}
\begin{code}
\begin{verbatim}
    on(p) {
      ...
    }
\end{verbatim}
\end{code}
\end{displaymath}
changes the active group to $\mbox{\tt p}$ inside its body.
The associated rule is:
\begin{usageRule}
The construct
\begin{displaymath}
\begin{code}
\begin{verbatim}
    on(p) { ...  }
\end{verbatim}
\end{code}
\end{displaymath}
can only appear at a point in the program where
{\tt p} is contained in the active process group.
\end{usageRule}
In other words, an {\tt on} construct cannot add any new process to
the active group.

Now, if the current active group
is {\tt p}, executing the construct construct
\begin{displaymath}
\begin{code}
\begin{verbatim}
    at(i = x [n]) {
      ...
    }
\end{verbatim}
\end{code}
\end{displaymath}
or
\begin{displaymath}
\begin{code}
\begin{verbatim}
    overall(i = x for l : u : s) {
      ...
    }
\end{verbatim}
\end{code}
\end{displaymath}
will change the active group to $\mbox{\tt p} / \mbox{\tt i}$ inside
the bodies of the constructs.

The expression $\mbox{\tt p} / \mbox{\tt i}$ is only well defined
if the location {\tt i} belongs to a range distributed over a dimension
of {\tt p}%
\footnote{Alternatively {\tt x} can be a collapsed range, in which
case $\mbox{\tt p} / \mbox{\tt i}$ is defined to be equal to {\tt p}.}%
.  So we can conclude that:
\begin{usageRule}
\label{rule:atOrOver}
Unless {\tt x} is a collapsed range, a control construct
\begin{displaymath}
\begin{code}
\begin{verbatim}
    at(i = x [n]) { ... }
\end{verbatim}
\end{code}
\end{displaymath}
or
\begin{displaymath}
\begin{code}
\begin{verbatim}
    overall(i = x for l : u : s) { ... }
\end{verbatim}
\end{code}
\end{displaymath}
can only appear at a point in the program where the process dimension
of {\tt x} occurs in the dimension set of the active process group.
\end{usageRule}
As an example of the application of this rule, notice that the the
dimension set of the restricted group $\mbox{\tt p} / \mbox{x[n]}$
certainly {\em does not} include the process dimension associated
with {\tt x}.  So one of the implications of rule \ref{rule:atOrOver}
is that we should never expect to see exactly the two constructs above
nested thus:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    at(i = x [n])
      overall(i = x for l : u : s) {     // error!
        ...
      }
\end{verbatim}
\end{code}
\end{displaymath}
This is good, because the outer construct already restricts
control to a single coordinate value, and it surely doesn't make sense
to try distributing control across all coordinates of the same
process dimension {\em inside} that construct%
\footnote{Strictly speaking this nesting is legal (but pointless) if
{\tt x} is collapsed.}%
.

\section{Rules for distributed array constructors}

There are several restrictions on distributed array constructor,
which we will group together in:
\begin{usageRule}
The distributed array constructor expression
\begin{displaymath}
  \mbox{\tt new } T \mbox{\tt [[}e_0\mbox{\tt , }\ldots
                    \mbox{\tt , }e_r\mbox{\tt , }\ldots
                    \mbox{\tt ]] on }p
\end{displaymath}
can only appear in a context where the distribution group, $p$,
is contained in the currently active process group.  If $e_r$ is a
(non-collapsed) range object, its process dimension must belong to
the dimension set of $p$.  No two range objects in ${e_0, \ldots,
e_r, \ldots}$ can be distributed over {\em the same} process dimension
of $p$.
\end{usageRule}
If the ``{\tt on }$p$'' clause is omitted, we identify the distribution
group, $p$, with the active process group, and the remaining conditions
must still apply.

%{\em [Worried that the specialized definition of ``subgroup'' might
%lead to confusion with the ``contains'' relation---which means set
%inclusion, with groups regarded as sets of processes.]}

\section{Rules for access to distributed array elements}
\label{sec:elementAccessRules}

First we will collect together rules for distributed array element
references implied or informally stated in \ref{sec:locations} and
subsequent sections.  The first rule is part of the static semantics
of the language---it can be enforced by the type checker:
\begin{usageRule}
If $a$ is a distributed array, then in the element reference
\begin{displaymath}
  a \mbox{\tt [} e_0 \mbox{\tt ,}\ldots
    \mbox{\tt ,} e_r \mbox{\tt ,}\ldots
    \mbox{\tt ]}
\end{displaymath}
the expression $e_r$ is either an integer expression---allowed only if
the corresponding dimension of $a$ has the sequential attribute---or a
(possibly shifted) distributed index.
\end{usageRule}
Depending on whether $e_r$ is in fact an integer, an index
declared in an {\tt at} construct, or an index declared in an {\tt overall}
construct, exactly one of the following three ``run-time''%
\footnote{Of course there is nothing to prevent a compiler applying
these checks at compile-time if it can do so.}
rules applies.  To simplify the discussion, we first ignore ghost regions.
The three rules are: either
\begin{usageRule}
\label{rule:integerSubscript}
In the distributed array element reference:
\begin{displaymath}
  a \mbox{\tt [} e_0 \mbox{\tt ,}\ldots
    \mbox{\tt ,} e_r \mbox{\tt ,}\ldots
    \mbox{\tt ]}
\end{displaymath}
if $e_r$ is an integer, its value must lie in the range
\begin{displaymath}
0 \le e_r < x \mbox{\tt .size()}
\end{displaymath}
where $x = a \mbox{\tt .rng(} r \mbox{\tt )}$.  
\end{usageRule}
or
\begin{usageRule}
\label{rule:atSubscript}
The distributed array element reference in:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$    at($$i$\verb$ = $$x$\verb$ [$$n$\verb$]) {$ \\
\verb$      ... $$a \mbox{\tt [}e_0\mbox{\tt ,} \ldots
                    \mbox{\tt ,} e_{r - 1}
                    \mbox{\tt ,} i
                    \mbox{\tt ,} e_{r + 1} \mbox{\tt ,} \ldots
                    \mbox{\tt ]}$ \verb$ ...$ \\
\verb$    }$
\end{tabbing}
\end{bigCode}
\end{displaymath}
is allowed if and only if
\begin{enumerate}
\item
The expression $a$ is invariant in the {\tt at} construct%
\footnote{
The interpretation of ``invariant'' will
be discussed further in section \ref{sec:subscriptChecking}.
Actually this condition is more important for the case
of an index subscript declared by an {\tt overall} (next rule).  
It is imposed here to preserve the semantic relation between {\tt at}
and {\tt overall} specified in \ref{sec:locations}.
}%
.

\item
The location $x \mbox{\tt [} n \mbox{\tt ]}$
is an element of the range $a \mbox{\tt .rng(} r \mbox{\tt )}$%
\footnote{This statement needs some interpretation, because
locations in certain ranges may be identified with locations in
others.  For example, locations in a subrange will be identified with
the matching locations of the parent range.  In fact it is possible for
two independently created ranges to be considered ``aligned'', in which
case their locations will be identified.  In general this will happen
if the two ranges are are distributed over exactly the same process
dimension and they have sufficiently similar distribution formats.
``Sufficiently similar'' usually means the distribution formats should
be structurally identical, but their is even some leeway here.  In
particular locations in an {\tt ExtBlockRange} can be identified with
the corresponding locations of a {\tt BlockRange} if the ranges have
the same extent and process dimension.  The {\tt Range} class has
includes methods such as {\tt isAligned} that can be used to determine if two
ranges are aligned, and thus logically share locations.}%
.
\end{enumerate}
\end{usageRule}
or
\begin{usageRule}
\label{rule:overallSubscript}
The distributed array element reference in:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$    overall($$i$\verb$ = $$x$\verb$ for $$l$\verb$ : $$u$\verb$ : $$s$\verb$) {$ \\
\verb$      ... $$a \mbox{\tt [}e_0\mbox{\tt ,} \ldots
                    \mbox{\tt ,} e_{r - 1}
                    \mbox{\tt ,} i
                    \mbox{\tt ,} e_{r + 1} \mbox{\tt ,} \ldots
                    \mbox{\tt ]}$ \verb$ ...$ \\
\verb$    }$
\end{tabbing}
\end{bigCode}
\end{displaymath}
is allowed if and only if
\begin{enumerate}
\item
The expression $a$ is invariant in the {\tt overall} construct.
\item
All locations in $x \mbox{\tt [} l \mbox{\tt :} u \mbox{\tt :} s \mbox{\tt ]}$
are of elements $a \mbox{\tt .rng(} r \mbox{\tt )}$.
\end{enumerate}
\end{usageRule}
A subtle and important point to appreciate is that rules
\ref{rule:atSubscript} and \ref{rule:overallSubscript} are statements
about the {\tt at} and {\tt overall} constructs as a whole, not about the array accesses
in isolation.  They apply unconditionally to any access that appears
textually inside the constructs, even if some conditional test in the
body of the construct might prevent those accesses from actually being
executed.  This is very important because it allows any associated
run-time checking code to be lifted outside the constructs, and in
particular to be lifted outside the local loops implied by an {\tt overall}.

\subsection{Changes for ghost regions}

If the array $a$ appearing in rule \ref{rule:integerSubscript} has
ghost regions%
\footnote{See section \ref{sec:ghostSplitting} for
an example of a sequential dimension with ghost regions.}%
, the range of allowed subscripts is changed to
\begin{displaymath}
- x \mbox{\tt .loExtension()} \le e_r <
x \mbox{\tt .size()} + x \mbox{\tt .hiExtension()}
\end{displaymath}
If the array appearing in rule \ref{rule:atSubscript} has
ghost regions, the index subscript may be shifted:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$    at($$i$\verb$ = $$x$\verb$ [$$n$\verb$]) {$ \\
\verb$      ... $$a \mbox{\tt [}e_0\mbox{\tt ,} \ldots
                    \mbox{\tt ,} e_{r - 1}
                    \mbox{\tt ,} i \pm d
                    \mbox{\tt ,} e_{r + 1} \mbox{\tt ,} \ldots
                    \mbox{\tt ]}$ \verb$ ...$ \\
\verb$    }$
\end{tabbing}
\end{bigCode}
\end{displaymath}
and the following requirement is added:
\begin{enumerate}
\setcounter{enumi}{2}
\item
The expression $\pm d$ is in the range
\begin{displaymath}
- a \mbox{\tt .rng(} r \mbox{\tt )} \mbox{\tt .loExtension()} \le \pm d \le a \mbox{\tt .rng(} r \mbox{\tt )} \mbox{\tt .hiExtension()}
\end{displaymath}
\end{enumerate}
Rule \ref{rule:overallSubscript} is modified in a completely
analogous way if the array appearing there has ghost regions.


\subsection{A final rule for array element access}

The rules on subscripts given in the last two subsections go a long
way towards ensuring a crucial requirement of HPJava, namely that
a process may only access locally held array elements.
There are still odd
cases---typically involving array sections---where those rules are insufficient.
Consider the pathological example of Figure \ref{fig:accessError}.
The subscripts on the element reference {\tt b [j]} are legal---{\tt j}
is certainly a location in {\tt b.rng(0)} (which is equal to {\tt y}).
But, as illustrated, the section
{\tt b} is localized to $\mbox{\tt p} / \mbox{\tt x[0]}$---the top row
of processes in the figure---whereas the {\tt at} construct
specifies that the element assignments are performed in the group
$\mbox{\tt p} / \mbox{\tt x[N-1]}$---the bottom row of processes.

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
                Procs2 p = new Procs2(P, P) ;
                on(p) {
                  Range x = new BlockRange(N, p.dim(0)) ;
                  Range y = new BlockRange(N, p.dim(1)) ;

                  float [[-,-]] a = new float [[x, y]] ;

                  float [[-]] b = a [[0, :]] ;

                  at(i = x [N - 1])
                    overall(j = y for :)
                      b [j] = j` ;               // error!
                }


\end{verbatim}
\end{code}
\end{displaymath}
%\centerline{\psfig{figure=accessError.eps,width=4.5in}}
\centerline{\includegraphics[width=4.5in]{accessError.eps}}
\caption{Access error discussed in text.  Section {\tt b} is shaded area at
top.  The at construct restricts control to the bottom row of
processes.\label{fig:accessError}}
\end{figure*}

This kind of error can be excluded by the following rule:
\begin{usageRule}
\label{rule:accessRule}
An element reference in an array $a$ can only be made by a process
that is contained in the distribution group $a \mbox{\tt .grp()}$.
\end{usageRule}
The error above is now exposed, because the distribution group of {\tt b}
is $\mbox{\tt p} / \mbox{\tt x[0]}$.  This does not
contain the active process group inside the {\tt overall} construct, namely
$\mbox{\tt p} / \mbox{\tt x[N-1]} / \mbox{\tt j}$.  So
the processes executing the array access fail to meet the criterion
of rule \ref{rule:accessRule}.

\section{A recommendation for updating variables}
\label{sec:canonical}

Suppose the distribution group of $a$ is $p$, and the list of subscripts
$e_0, \ldots, e_r, \ldots$ in the element reference
\begin{displaymath}
  a \mbox{\tt [}e_0\mbox{\tt , }\ldots
    \mbox{\tt , }e_r\mbox{\tt , }\ldots
    \mbox{\tt ]}
\end{displaymath}
includes locations $i, j, \ldots$, then
the {\em home group} of the array element is defined to be
\begin{displaymath}
    p / i / j / \ldots
\end{displaymath}
If the array has a replicated distribution this group may contain
several processes; otherwise it contains a single process.

The definition of the {\em home group} of a distributed array element
can be extended to other kinds of variable: the home group of a
variable (not a distributed array element) is simply the active process group
at the point where the variable is declared.

A good rule of thumb for updating variables in general is
\begin{recommendation}
A variable should only be updated when the active process
group is identical to the home group of the variable.
\end{recommendation}
If this rule is followed rigorously throughout a program (and if
different processes only ever diverge in behaviour through dependencies
on values of global indexes in {\tt overall} constructs, or values of locally
held program variables) it has the interesting result that all
variables remain {\em coherent}.  A variable is coherent if, at
corresponding stages of execution of an SPMD program, all processes in
the home group of a variable hold identical values in their
local copies of the variable.

There are many places in HPJava programs where variables are {\em
required} to be coherent.  This is particularly true of arguments to
collective operations.  There are other places where it can be
convenient to relax the coherence rule, which is one reason why it is only
advisory (another reason is that it is relatively expensive to enforce this rule
by runtime checks).

We call the style of programming in which all variables are held coherent
the {\em canonical HPspmd style}.  Out of the examples given
so far in this document, the only one that doesn't follow canonical
style is the Monte Carlo program of Figure \ref{fig:ising}.  In that
program the variable {\tt rand} has home group {\tt p}, but it is
updated inside nested {\tt overall} constructs, where the active process
group is $\mbox{\tt p} / \mbox{\tt i} / \mbox{\tt j}$.  Also,
the initialization of {\tt rand} involves a dependency on the {\tt
crd()} method of {\tt Dimension}, which is intrinsically incoherent.
All other variables in all other algorithmic examples are coherent%
\footnote{There is a short example in section \ref{sec:processes}
that uses the {\tt crd()} inquiry, and therefore isn't canonical.}% 
.  The canonical HPspmd style has a special affinity with the pure data
parallel programming style of languages like HPF.

\chapter{A Distributed Array Communication Library\label{sec:library}}

Many of the examples in this article use a communication library called
Adlib.  This library is not supposed to have a uniquely special
status so far as the HPJava language definition is concerned.  Adlib
was developed independently of the HPJava project, to support HPF
translation.  Eventually HPJava bindings for other communication
libraries will be needed.  For example, the Adlib library does not
provide the one-sided communication functionality of libraries like the
Global Arrays toolkit; it doesn't provide optimized numerical
operations on distributed arrays like those in ScaLAPACK; it does
not provide highly optimized collective operations for irregular
patterns of array access, like those in CHAOS.  All these libraries
(and others) work with distributed arrays more or less similar to
HPJava distributed arrays.  We hope that bindings to these libraries,
or functionally similar libraries, will be made available in HPJava.
For now, this section summarizes essential features of the HPJava
binding to Adlib.

\section{Regular collective communications}

There are three main families of collective operation in Adlib: regular
communications, reduction operations, and irregular communications.

The regular communications are exemplified the operations {\tt shift},
{\tt cshift}, {\tt writeHalo} and {\tt remap}, introduced in earlier
sections.  The last of these, {\tt remap}, is a very
characteristic example.  The {\tt remap} function takes two distributed
array arguments---a source array and a destination.  These two
arrays must have the same size and shape%
\footnote{The {\em shape} of a distributed array is the list of its
extents, {\tt (a.rng(0).size(), a.rng(1).size(), ...)}.}
but they can have any, unrelated, distribution formats.  The effect of
the operation is to copy the values of the elements in the source array
to the corresponding elements in the destination array, performing any
communications required to do that.  If the destination array has {\em
replicated} mapping, the {\tt remap} operation will broadcast source
values to all copies of the destination array elements.

The {\tt remap} function is a static member of the {\tt Adlib} class.
Like most of the functions in Adlib, the {\em remap} function is
overloaded to apply to various ranks and types of array:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    void remap(int    [[-]] destination, int    [[-]] source) ;
    void remap(float  [[-]] destination, float  [[-]] source) ;
    void remap(double [[-]] destination, double [[-]] source) ;
    ...
    void remap(int    [[-,-]] destination, int    [[-,-]] source) ;
    void remap(float  [[-,-]] destination, float  [[-,-]] source) ;
    void remap(double [[-,-]] destination, double [[-,-]] source) ;
    ...
    void remap(int    [[-,-,-]] destination, int    [[-,-,-]] source) ;
    void remap(float  [[-,-,-]] destination, float  [[-,-,-]] source) ;
    void remap(double [[-,-,-]] destination, double [[-,-,-]] source) ;
    ...
\end{verbatim}
\end{code}
\end{displaymath}
and scalars:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    void remap(int    [[]] destination, int    [[]] source) ;
    void remap(float  [[]] destination, float  [[]] source) ;
    void remap(double [[]] destination, double [[]] source) ;
    ...
\end{verbatim}
\end{code}
\end{displaymath}
Currently the element-type overloading includes all Java {\em primitive}
types.  Later Adlib will be extended to support {\tt Object} types.

There are four preconditions for a call to {\tt remap}:
\begin{enumerate}
\item
All processes in the active process group must make the call, and they
must pass coherent arguments---for each argument, all
processes pass local references to logically the same distributed
array.
\item
As mentioned above, the source and destination arrays should have
the same shape and element types.
\item
The arrays {\tt source} and {\tt destination} must not overlap---no
element of {\tt source} must be an alias for an element of {\tt destination}.
This is only an issue if both arguments are sections of the same array.
\item
Both arguments must be {\em fully contained} in the active process group.
\end{enumerate}
By definition, an array is ``fully contained'' if its distribution
group is contained in the active process group.  So the
requirement is that every copy of every element of the array is held on
one of the processors engaging in the collective operation.

Most of the functions in Adlib have a similar set of
preconditions---all operations are called collectively with coherent
arguments, input and output arrays should never overlap, and array
arguments must always be fully contained in the active group.  The last
requirement is probably the easiest to overlook.  Consider the example
of section \ref{sec:replicated}, Figure \ref{fig:generalMatmul}.  An
easy mistake would be to put the calls to {\tt remap} {\em inside} the
following {\tt on} construct.  This is an error, because there is no
guarantee that distribution groups of {\tt a} and {\tt b} are contained
in the distribution group, {\tt p}, of {\tt c}.  The function {\tt
matmul} is supposed to work for arguments with any, unrelated,
distribution format.  The Adlib library includes runtime checks for
containment of arrays.  If an argument is not fully contained, an
exception occurs.

So long as the rule on containment is observed, Adlib calls can
be made freely inside distributed control constructs, including
the parallel loop, {\tt overall}.  If, for example, we want to ``skew''
an array---shift rows in the {\tt y} direction by an amount
that depends on the {\tt x} index---we can do something like
\begin{displaymath}
\begin{code}
\begin{verbatim}
  on(p) {
    int [[-,-]] a = new int [[x, y]], b = new int [[x, y]] ;

    overall(i = x for :)
      Adlib.shift(b [[i, :]], a [[i, :]], i`) ;
  }
\end{verbatim}
\end{code}
\end{displaymath}
The section arguments of {\tt shift} have distribution group 
$\mbox{\tt p} / \mbox{\tt i}$, which is identical to the active process
group at this point, so the arguments are fully contained.  A slightly
more complicated example involving {\tt dotProduct} was given earlier
in section \ref{sec:arrayRestriction}, Figure \ref{fig:3dMatmul}.

A prototype of the {\tt shift} function was given in section
\ref{sec:parallel}.  In general we have
\begin{displaymath}
\begin{code}
\begin{tabbing}
\verb$    void shift($$T$\verb$ [[-]] destination, $$T$\verb$ [[-]] source,$ \\
\verb$               int shiftAmount) ;$ \\
\verb$    void shift($$T$\verb$ [[-,-]] destination, $$T$\verb$ [[-,-]] source,$ \\
\verb$               int shiftAmount, int dimension) ;$ \\
\verb$    void shift($$T$\verb$ [[-,-,-]] destination, $$T$\verb$ [[-,-,-]] source,$ \\
\verb$               int shiftAmount, int dimension) ;$ \\
\verb$    ...$
\end{tabbing}
\end{code}
\end{displaymath}
where $T$ stands as a shorthand for any primitive type of Java.  
The {\tt dimension} argument is in the range $0, \ldots, R - 1$ where
$R$ is the rank of the arrays.  It selects the array dimension in which
the shift occurs.  The {\tt shiftAmount} argument, which may be
negative, specifies the amount and direction of the shift.
Again the source and destination arrays must have the same shape, but
now there is an extra precondition---they must also be identically
aligned.  That is, their distribution groups must be identical and all
their ranges must be identical or satisfy the {\tt isAligned} test.  By
design, {\tt shift} implements a simpler pattern of communication than
general {\tt remap}.  The alignment relation allows a more efficient
implementation.  The library includes runtime checks on alignment
relations between arguments, where these are required.

The {\tt shift} operation discards values from {\tt source} that
are moved past the edge of {\tt destination}.  At the other end of the
range, elements of {\tt destination} that are not targetted by
elements from {\tt source} are unchanged from their input value.
The operation {\tt cshift} is essentially identical to {\tt shift}
except that it implements a circular shift.

The function {\tt writeHalo} is applied to distributed arrays that have
ghost regions.  It updates those regions.  The simplest versions have
prototypes
\begin{displaymath}
\begin{code}
\begin{tabbing}
\verb$    void writeHalo($$T$\verb$ [[-]] a) ;$ \\
\verb$    void writeHalo($$T$\verb$ [[-,-]] a) ;$ \\
\verb$    void writeHalo($$T$\verb$ [[-,-,-]] a) ;$ \\
\verb$    ...$
\end{tabbing}
\end{code}
\end{displaymath}
We can distinguish between the locally held {\em physical segment} of an array
and the surrounding {\em ghost region}, which is used to cache local copies
of remote elements.  The effect of {\tt writeHalo}
is to overwrite the ghost region with
values from processes holding the corresponding elements in their physical
segments.  

A more general form of {\tt writeHalo} allows to specify
that only a subset of the available ghost area is to be updated, and
to select circular wraparound for updating ghost cells
at the extreme ends of the array, if desired.
\begin{displaymath}
\begin{code}
\begin{tabbing}
\verb$    void writeHalo($$T$\verb$ [[-]] a, int wlo, int whi, int mode) ;$ \\
\verb$    void writeHalo($$T$\verb$ [[-,-]] a, int wlo [], int whi [], int [] mode) ;$ \\
\verb$    void writeHalo($$T$\verb$ [[-,-,-]] a, int wlo [], int whi [], int [] mode) ;$ \\
\verb$    ...$
\end{tabbing}
\end{code}
\end{displaymath}
The integer vectors are all of length $R$, the rank of the argument {\tt
a}.  The values {\tt wlo} and {\tt whi} specify the widths at upper and lower ends of
the bands to be updated (these values must be less than or equal to the
widths of the actual ghost areas on the array).  The {\tt
mode} values define for each dimension whether to update in the normal way,
leaving ghost edges at extreme edges of the arrays unwritten (value
should be {\tt WriteHalo.EDGE}), whether to update using circular
wraparound ({\tt WriteHalo.CYCL}), or whether to not update any ghost
regions in this dimension at all ({\tt WriteHalo.NONE}, equivalent to
setting the corresponding elements of {\tt wlo}, {\tt whi} to zero).

Operation of {\tt writeHalo} is visualized in figure \ref{fig:writeHalo}.  

\begin{figure}[btp]
%\centerline{\psfig{figure=WriteHalo.eps,height=3in}}
\centerline{\includegraphics[width=3in]{WriteHalo.eps}}
\caption{Illustration of the effect of executing the {\tt writeHalo}
function.\label{fig:writeHalo}}
\end{figure}

\section{Reductions}

Reduction operations take one or more distributed arrays as input.
They combine the elements to produce one or more scalar values, or
arrays of lower rank.  Adlib provides a large set of reduction
operations, supporting the many kinds of reduction available in as
``intrinsic functions'' in Fortran.  Here we mention only a few of
the simplest reductions.

The {\tt maxval} operation simply returns the maximum of all elements of
an array.  It has prototypes
\begin{displaymath}
\begin{code}
\begin{tabbing}
\verb$    $$T$\verb$ maxval($$T$\verb$ [[-]] a) ;$ \\
\verb$    $$T$\verb$ maxval($$T$\verb$ [[-,-]] a) ;$ \\
\verb$    $$T$\verb$ maxval($$T$\verb$ [[-,-,-]] a) ;$ \\
\verb$    ...$
\end{tabbing}
\end{code}
\end{displaymath}
The result is broadcast to the active process group, and
returned by the function.  Other reduction operations with similar
interfaces are {\tt minval}, {\tt sum} and {\tt product}.  Of these
{\tt minval} is minimum value, {\tt sum} adds the elements of {\tt
a} in an unspecified order, and {\tt product} multiplies them.

The function {\tt dotProduct} used in some earlier examples is also
logically a reduction, but it takes two one-dimensional arrays
as arguments and returns
their scalar product---the sum of pairwise products of elements.
The situation with element types is complicated
because the types of the two arguments needn't be identical.
If they are different, standard Java binary numeric promotions are
applied---for example if the dot product of an {\tt int} array with
a {\tt float} array is a {\tt float} value.  Some of the
prototypes are
\begin{displaymath}
\begin{code}
\begin{verbatim}
    int    dotProduct(int   [[-]] a, int    [[-]] b) ;
    float  dotProduct(int   [[-]] a, float  [[-]] b) ;
    double dotProduct(int   [[-]] a, double [[-]] b) ;
    float  dotProduct(float [[-]] a, int    [[-]] b) ;
    float  dotProduct(float [[-]] a, float  [[-]] b) ;
    double dotProduct(float [[-]] a, double [[-]] b) ;
    ...
\end{verbatim}
\end{code}
\end{displaymath}
The arguments must have the same shape and must be aligned.  As usual the
result is broadcast to all members of the active process group.

The function {\tt broacast} is not actually a reduction, but it has
some features in common with other functions discussed in this section.
The prototype is
\begin{displaymath}
\begin{code}
\begin{tabbing}
\verb$    $$T$\verb$ broadcast($$T$\verb$ [[]] s) ;$ \\
\end{tabbing}
\end{code}
\end{displaymath}
It takes a scalar (rank-0 distributed array) as argument and broadcasts
the element value to all processes of the active process group.  Typically
it is used in conjunction with a scalar section to broadcast
an element of a general array, as in this fragment:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    int [[-,-]] a = new int [[x, y]] ;

    int n = 3 + Adlib.broadcast(a [[10, 10]]) ;
\end{verbatim}
\end{code}
\end{displaymath}

\section{Irregular collective communications}

Adlib has some support for irregular communications in the form
of collective {\tt gather} and {\tt scatter} operations.
The simplest form of the gather operation for one-dimensional
arrays has prototypes
\begin{displaymath}
\begin{code}
\begin{tabbing}
\verb$    void gather($$T$\verb$ [[-]] destination, $$T$\verb$ [[-]] source, int [[-]] subscripts) ;$
\end{tabbing}
\end{code}
\end{displaymath}
The {\tt subscripts} array should have the same shape as, and be
aligned with, the {\tt destination} array.  In pseudocode, the {\tt gather}
operation is equivalent to
\begin{displaymath}
\begin{code}
\begin{tabbing}
\verb$    for all $$i$\verb$ in $$\{0, \ldots, N-1\}$\verb$ in parallel do$ \\
\verb$      destination [$$i$\verb$] = source [subscripts [$$i$\verb$]] ;$
\end{tabbing}
\end{code}
\end{displaymath}
where $N$ is the size of the {\tt destination} (and {\tt subscripts}) array.
If we are implementing a parallel algorithm that involves a stage like
\begin{displaymath}
\begin{code}
\begin{tabbing}
\verb$    for all $$i$\verb$ in $$\{0, \ldots, N-1\}$\verb$ in parallel do$ \\
\verb$      a [$$i$\verb$] = b [${\em fun(i)}\verb$] ;$
\end{tabbing}
\end{code}
\end{displaymath}
where {\em fun} is an arbitrary function, it can be expressed in HPJava as
\begin{displaymath}
\begin{code}
\begin{tabbing}
\verb$    int [[-]] tmp = new int [[x]] on p ;$ \\
\verb$    on(p)$ \\
\verb$      overall(i = x for :)$ \\
\verb$        tmp [i] = ${\em fun(}\verb$i${\em )} ; \\
\\
\verb$    Adlib.gather(a, b, tmp) ;$
\end{tabbing}
\end{code}
\end{displaymath}
where {\tt p} and {\tt x} are the distribution group and range of {\tt a}.
The source array may have a completely unrelated mapping.

The one-dimensional case generalizes to give a rather complicated
family of prototypes for multidimensional arrays:
\begin{displaymath}
\begin{code}
\begin{tabbing}
\verb$    void gather($$T$\verb$ [[-]] destination, $$T$\verb$ [[-]] source,$ \\
\verb$                int [[-]] subscripts) ;$ \\
\verb$    void gather($$T$\verb$ [[-,-]] destination, $$T$\verb$ [[-]] source,$ \\
\verb$                int [[-,-]] subscripts) ;$ \\
\verb$    void gather($$T$\verb$ [[-,-,-]] destination, $$T$\verb$ [[-]] source,$ \\
\verb$                int [[-,-,-]] subscripts) ;$ \\
\verb$    ...$ \\
\verb$    void gather($$T$\verb$ [[-]] destination, $$T$\verb$ [[-,-]] source,$ \\
\verb$                int [[-]] subscripts1, int [[-]] subscripts2) ;$ \\
\verb$    void gather($$T$\verb$ [[-,-]] destination, $$T$\verb$ [[-,-]] source,$ \\
\verb$                int [[-,-]] subscripts1, int [[-,-]] subscripts2) ;$ \\
\verb$    void gather($$T$\verb$ [[-,-,-]] destination, $$T$\verb$ [[-,-]] source,$ \\
\verb$                int [[-,-,-]] subscripts1, int [[-,-,-]] subscripts2) ;$ \\
\verb$    ...$ \\
\verb$    ...$
\end{tabbing}
\end{code}
\end{displaymath}
The complexity arises because now that the source and destination
arrays can have different ranks.  The pattern is that the subscript arrays
have the same and alignment shape as the destination arrays.  The
{\em number} of subscript arrays is equal to the rank of the source array.
As an example, the last of the prototypes enumerated above
behaves like
\begin{displaymath}
\begin{code}
\begin{tabbing}
\verb$    for all $$i$\verb$ in $$\{0, \ldots, L-1\}$\verb$ in parallel do$ \\
\verb$      for all $$j$\verb$ in $$\{0, \ldots, M-1\}$\verb$ in parallel do$ \\
\verb$         for all $$k$\verb$ in $$\{0, \ldots, N-1\}$\verb$ in parallel do$ \\
\verb$           destination [$$i$\verb$, $$j$\verb$, $$k$\verb$] = source [$\=\verb$subscripts1 [$$i$\verb$, $$j$\verb$, $$k$\verb$],$ \\
                  \>\verb$subscripts2 [$$i$\verb$, $$j$\verb$, $$k$\verb$]] ;$
\end{tabbing}
\end{code}
\end{displaymath}
where $(L, M, N)$ is the shape of {\tt destination} array.

The basic {\tt scatter} function has very similar prototypes, but
the names {\tt source} and {\tt destination} are switched.  The
one-dimensional case is
\begin{displaymath}
\begin{code}
\begin{tabbing}
\verb$    void scatter($$T$\verb$ [[-]] source, $$T$\verb$ [[-]] destination,$ \\
\verb$                 int [[-]] subscripts) ;$ \\
\end{tabbing}
\end{code}
\end{displaymath}
and it behaves like
\begin{displaymath}
\begin{code}
\begin{tabbing}
\verb$    for all $$i$\verb$ in $$\{0, \ldots, N-1\}$\verb$ in parallel do$ \\
\verb$      destination [subscripts [$$i$\verb$]] = source [$$i$\verb$] ;$
\end{tabbing}
\end{code}
\end{displaymath}

\section{Schedules}

In general the collective communication
functions introduced in the last few sections involve two phases:  an
{\em inspector} phase in which the arguments are analysed to determine
what communications and local copies will be needed to complete the
operation, and an {\em executor} phase in which the schedule of these
data transfers is actually performed.
In iterative algorithms, it often happens that exactly the same
communication pattern is repeated many times over.  In this case it
is wasteful to repeat the inspector phase in every iteration, because
the data transfer schedule will be the same every time.

Adlib provides a class of {\em schedule objects} for each of its
communication functions.  The classes generally have the same names as
the static functions, with the first letter capitalized (the name
may also be extended with a result type).  Each class has a series of
constructors with arguments identical to the instances of the
function.  Every schedule class has one public method with no arguments
called {\tt execute}, which executes the schedule.

Using {\tt WriteHalo} and {\tt Maxval} schedules, the main loop of the
red-black relaxation program from section \ref{sec:ghostRegions},
Figure \ref{fig:redBlack} could be rewritten as in Figure
\ref{fig:scheduleRedBlack}.

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
    WriteHalo writeHalo = new WriteHalo(u) ;
    MaxvalFloat maxval = new MaxvalFloat(r) ;

    do {
      for(int parity = 0 ; parity < 2 ; parity++) {

        writeHalo.execute() ;

        overall(i = x for 1 : N - 2)
          overall(j = y for 1 + (i` + parity) % 2 : N - 2 : 2) {
            float newA ;

            newA = 0.25 * (a [i - 1, j] + a [i + 1, j] +
                           a [i, j - 1] + a [i, j + 1]) ;

            r [i, j] = Math.abs(newA - a [i, j]) ;
            a [i, j] = newA ;
          }
      }

    } while(maxval.execute() > EPS)) ;
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Red-black relaxation, re-using communication schedules.%
\label{fig:scheduleRedBlack}}
\end{figure*}

{\em [Need to document {\tt HPspmd.copy()} somewhere.]}


\chapter{Low level SPMD programming}
\label{sec:lowLevelSPMD}

It often happens that some parts of a large parallel program cannot be
written efficiently in the pure data parallel style, using {\tt overall}
constructs to process all elements of distributed arrays on essentially
the same footing.  Sometimes, for efficiency, a process has to be more
``introspective''---it has to get down and do some procedure that combines
the locally held array elements in a non-trivial way.  The local results
may be combined with off-processor results in a separate step.

\section{An Example}

We will consider a fragment from a parallel {\em N-body} classical
mechanics problem.  As the name suggests, this problem is concerned with
the dynamics of a set of N interacting bodies.  The total force on each
body includes a contribution from all the other bodies in the system.
The size of this contribution depends on the position, $x$, of
the body experiencing the force, and the position, $y$, of the body
exerting it.  If the individual contribution is given by
$\mbox{\tt force(}x, y\mbox{\tt )}$, the net force on body $i$ is
\begin{displaymath}
\sum_{j} \mbox{force(}a_i, a_j\mbox{\tt )}
\end{displaymath}
where now $a_j$ is the position of the $j$th body.
The total force can be computed in parallel by the program given
in \ref{fig:data-Nbody}.  We repeatedly rotate a copy, {\tt b},
of the position vector, {\tt a}, using {\tt cshift}.
Every element in {\tt b} thus passes by every element in the fixed
vector {\tt a}, and contributions to the force are accumulated as we go.
The approach is similar to the pipelined matrix multiplication in
Figure \ref{fig:pipedMatmul}.

\begin{figure}[t]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs1 p = new Procs1(P) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;

    float [[-]] f = new float [[x]], a = new float [[x]],
                b = new float [[x]] ;

    ... initialize `a' ...

    overall(i = x for :) {
      f [i] = 0.0 ;
      b [i] = a [i] ;
    }

    for(int s = 0 ; s < N ; s++) {

      overall(i = x for :)
        f [i] += force (a [i], b [i]) ;

      // cyclically shift `b' (by amount -1 in x dim)...

      Adlib.cshift(tmp, b, -1) ;
      HPspmd.copy(b, tmp) ;
    }
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Data parallel version of the N-body force computation.\label{fig:data-Nbody}}
\end{figure}


The trouble is that this involves $N$ small shifts (Figure
\ref{fig:dataParNBody}).  Calling out to the communication library so
many times (and copying a whole array so many times) is likely to produce
an inefficient program.

\begin{figure}[btp]
%\centerline{\psfig{figure=dataParNBody.eps,width=5in}}
\centerline{\includegraphics[width=5in]{dataParNBody.eps}}
\caption{Simple ``data parallel'' N-body force computation.  The array
{\tt b} is shifted one element to the right in each iteration.  Arrows define
element pairs combined in the iteration.\label{fig:dataParNBody}}
\end{figure}

In fact we can achieve an equivalent effect---passing the value of every
element by every other---if we do just $P$ iterations of an outer loop,
with each iteration shifting the {\em whole block} of locally held
elements of the moving copy to the neighbouring process
(Figure \ref{fig:taskParNBody}).

\begin{figure}[btp]
%\centerline{\psfig{figure=taskParNBody.eps,width=5in}}
\centerline{\includegraphics[width=5in]{taskParNBody.eps}}
\caption{Efficient N-body force computation.  The array
{\tt b} is shifted one {\em block} to the right in each iteration. 
Arrows define element pairs combined in the iteration.\label{fig:taskParNBody}}
\end{figure}

We can express the second algorithm straightforwardly enough in the
language defined so far, but the price is that we have to change
the way the distributed arrays are represented in the program.  

One legitimate way to express the algorithm is in
a direct SPMD message-passing style.  Example code is given in
Figure \ref{fig:mpi-Nbody}.  The local block size is
{\tt B}, so the value of {\tt N} is $\mbox{\tt P} \times \mbox{\tt B}$.
For the sake of being concrete, we have used the methods {\tt Rank()} and
{\tt Sendrecv\_replace()} from the {\em mpiJava} \cite{mpiJava} binding of
the Message Passing Interface, MPI \cite{MPI}.  Figure \ref{fig:mpi-Nbody}
is a valid SPMD Java program and thus a valid HPJava program.  Unfortunately
it is lives in a different universe of data structures and communication
functions from the parallel-array, collective-communication oriented
algorithms we have seen so far.  We need to build some bridge between
these two extreme styles of parallel programming.

\begin{figure}[t]
\begin{displaymath}
\begin{code}
\begin{verbatim}
    int myID = MPI.COMM_WORLD.Rank();

    float [] f = new float [B], a = new float [B], b = new float [B] ;

    ... initialize `a' ...

    for(int i = 0 ; i < B ; i++) {
      f[i] = 0.0 ;
      b[i] = a[i] ;
    }

    for(int s = 0 ; s < P ; s++) {

      for(int i = 0 ; i < B ; i++)  // B : local block size
        for(int j = 0 ; j < B ; j++)
          f [i] += force(a [i], b [j]) ;


      // cyclically shift `b'...

      int right = (myID + 1) % P, left = (myID + P - 1) % P ;

      MPI.COMM_WORLD.Sendrecv_replace(b, 0, B, MPI.FLOAT,
                                      right, 0, left, 0) ;
    }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{MPI version of the N-body force computation.\label{fig:mpi-Nbody}}
\end{figure}

One approach---perhaps not the most obvious, but quite natural in
HPJava---is to explicitly split the index space of the original
one-dimensional arrays across two dimensions: a distributed dimension
representing the process dimension itself, and a sequential dimension
explicitly representing the local block.  The code is given in Figure
\ref{fig:task-Nbody}.

\begin{figure}[t]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs1 p = new Procs1(P) ;
  Dimension d = p.dim(0) ;

  on(p) {

    float [[-,*]] f = new float [[d, B]], a = new float [[d, B]],
                  b = new float [[d, B]] ;   

    ... initialize `a' ...

    overall(i = d for :)
      for (int j = 0 ; j < B ; j++) {
        f [i, j] = 0.0 ;
        b [i, j] = a [i, j] ;
      }

    for(int s = 0 ; s < P ; s++) {

      overall(i = d for :)
        for(int j = 0 ; j < B ; j++)
          for(int k = 0 ; k < B ; k++)
            f [i, j] += force(a [i, j], b [i, k]) ;

      // cyclically shift `b' in `d' dim...

      float [[-,*]] tmp = new float [[d, B]] ;

      Adlib.cshift(tmp, b, 1, 0) ;
      HPspmd.copy(b, tmp) ;
    }
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Efficient HPJava version of the N-body force computation.\label{fig:task-Nbody}}
\end{figure}

Although there is only one element of {\tt d} associated with each
process, the rules of HPJava force us to explicitly subscript in the
associated array dimension.  This leads to some extra verbosity, but
this style of programming has some attractive features:
\begin{itemize}
\item
As a practical matter, the fact we are dealing with true HPJava
distributed arrays means we can continue to employ concise calls
to collective library functions like {\tt cshift()}, instead of
relatively clumsy functions like {\tt Sendrecv\_replace()}.
\item
More esoterically, the program follows the {\em canonical HPspmd
style}, described briefly in section \ref{sec:canonical}.  As defined
in that section, all variables are {\em coherent}.  The elements of
the arrays in the program of Figure \ref{fig:mpi-Nbody} are {\em not}
coherent, because they take different values in each process, although
their home group is the set of all processes.  
Respecting the canonical style may not have immediate practical
advantages, but it is somehow aesthetically pleasing.  In this
style there is no need for incoherent functions like {\tt Rank()} to get
the local process id---instead one uses global index values associated
with {\tt overall} constructs.
\end{itemize}

\clearpage

\section{Dimension Splitting}

This style goes some way toward forging a link between low-level SPMD
programming and the higher level data-parallel style of HPJava, but by
itself it doesn't help if we are presented with an {\em existing}
one-dimensional, block-distributed array, and required to do some low-level
processing on its blocks.  To allow for this situation,
the language is extended to support {\em dimension splitting}.
Dimension splitting is introduced as an extension of the
array section mechanism described at length in Chapter \ref{sec:sections}.
The extra syntactic baggage is minimal, but the repercussions are
quite far-reaching.

First we note that a particular element in a distributed array can
be identified in one of two ways.  It can be identified by giving a
{\em global} subscript in the distributed range, which is effectively
what we have done in HPJava in earlier chapters.  Alternatively it can be
identified by giving a process coordinate and a {\em local subscript}---a
subscript within the array segment held on the associated process.
Dimension splitting provides a way of accessing an element through
its process coordinate and local subscript, by allowing a distributed
dimension of an array to be temporarily viewed as {\em two} dimensions---a
coordinate dimension plus a local subscript dimension.

If the subscript in a particular dimension of a section expression
is the special symbol {\tt <>}, that dimension of the array is split.
Whereas a triplet subscript in a section expression yields one dimension
in the result array, a splitting subscript yields two---a distributed
dimension and a sequential dimension.  
The range of the distributed dimension is
the process dimension over which the original array dimension was distributed%
\footnote{Usually, but see Section \ref{sec:blockCyclic}.}%
; the local blocks of the original array are embedded in the the sequential
dimension of the result.  The two new dimensions appear consecutively
in the signature of the result array, distributed dimension first.

Now we can combine the examples from Figures
\ref{fig:data-Nbody} and \ref{fig:task-Nbody} of the last section.
The version in Figure \ref{fig:dimspl-Nbody} initally creates
the arrays as distributed, one-dimensional arrays, and uses this
convenient form to initialize them.  It then uses a split representation
of the same arrays to compute the force array.  
Note that, because {\tt as} and {\tt fs} are semantically {\em sections}
of {\tt a} and {\tt f}, they share common elements---they provide aliases
through which the same element variables can be accessed.  So when the
computation loop is complete, the vector of forces can be accessed again
through the one-dimensional array {\tt f}.  This is likely to be what
is needed in this case.

\begin{figure}[t]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs1 p = new Procs1(P) ;
  Dimension d = p.dim(0) ;

  on(p) {
    Range x = new BlockRange(N, d) ;

    float [[-]] f = new float [[x]], a = new float [[x]],
                b = new float [[x]] ;

    ... initialize `a' ...

    overall(i = x for :) {
      f [i] = 0.0 ;
      b [i] = a [i] ;
    }


    // split `x' dimensions:

    float [[-,*]] fs = f [[<>]], as = a [[<>]], bs = b [[<>]] ;   

    for(int s = 0 ; s < P ; s++) {

      overall(i = d for :)
        for(int j = 0 ; j < B ; j++)
          for(int k = 0 ; k < B ; k++)
            fs [i, j] += force(as [i, j], bs [i, k]) ;

      // cyclically shift `bs' in `d' dim...

      Adlib.cshift(tmp, bs, 1, 0) ;
      HPspmd.copy(bs, tmp) ;
    }
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Version of the N-body force computation using dimension splitting.\label{fig:dimspl-Nbody}}
\end{figure}

As a similar but slightly more complicated example, Figure
\ref{fig:dimsplPipedMatmul} contains an optimized version of the pipelined
matrix multiplication from Figure \ref{fig:pipedMatmul}.
Here the arithmetic is done in local blocks by a method
{\tt matmul},
which implements the matrix multiplication $\mbox{\tt c} = \mbox{\tt a}
\times {\tt b}$ on sequential two-dimensional arrays.
It could be written in elementary style as
\begin{displaymath}
\begin{code}
\begin{verbatim}
    void matmul(float [[*,*]] c, float [[*,*]] a, float [[*,*]] b) {

      int l = c.rng(0).size(), m = c.rng(1).size(), n = a.rng(1).size() ;

      for (int i = 0 ; i < l ; i++)
        for (int j = 0 ; j < m ; j++) {
          c [i, j] = 0.0 ;
          for (int k = 0 ; k < n ; k++)
            c [i, j] += a [i, k] * b [k, j] ;
        }
    }
\end{verbatim}
\end{code}
\end{displaymath}
or it could be an optimized library routine.  The operation
of the parallel algorithm for $\mbox{\tt P} = 2$ is illustrated in
\ref{fig:matmul1d}.

\begin{figure}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs1 p = new Procs1(P) ;
  Dimension d = p.dim(0) ;

  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;

    float [[-,*]] a = new float [[x, N]], c = new float [[x, N]] ;
    float [[*,-]] b = new float [[N, x]] ;

    ... initialize `a', `b'


    // split `x' dimensions:

    float as [[-,*,*]] = a [[<>, :]] ;
    float bs [[*,-,*]] = b [[:, <>]] ;
    float cs [[-,*,*]] = c [[<>, :]] ;

    for(int s = 0 ; s < P ; s++) {

      overall (i = d for :) {
        const int base = B * ((i` + s) % P) ;

        matmul(cs [[i, :, base : base + B - 1]],
                                as [[i, :, :]], bs [[:, i, :]]) ;
      }

      // cyclically shift down `bs' in `d' dim...

      float tmp [[*,-,*]] = new float [[N, d, B]] ;

      Adlib.cshift(tmp, bs, -1, 1) ;
      HPspmd.copy(bs, tmp) ;
    }
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Pipelined matrix multiplication program using dimension
splitting.\label{fig:dimsplPipedMatmul}}
\end{figure}

\begin{figure}[btp]
%\centerline{\psfig{figure=matmul1d.eps,width=5in}}
\centerline{\includegraphics[width=5in]{matmul1d.eps}}
\caption{Operation of efficient pipelined matrix computation.  The matrix
{\tt b} is shifted one block to the right in each iteration. 
\label{fig:matmul1d}}
\end{figure}

\clearpage

\section{Block Parameters}

\label{sec:blockParameters}

That is quite neat, but unfortunately it isn't the end of the story.
The examples of the previous section will only work properly if {\tt N}
is an exact multiple of the number of processes, {\tt P}, so that the
block size, {\tt B}, is identical in all processes.  In general we do
not want to limit ourselves to this case.

A few of the possibilities for mapping a 50-element, one-dimensional
array to 4 processes are illustrated in Figure \ref{fig:mapExamples}.
They presume the declarations:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    Procs p = new Procs1(4) ;
    Dimension d = p.dim(0) ;
\end{verbatim}
\end{code}
\end{displaymath}
In the first case the array {\tt a} is divided into four contiguous
blocks of sizes $(13,13,13,11)$.  In the second case the blocking is
different---$(13,13,12,12)$---and the formula for computing the global index
value is quite different.  The third case illustrates that {\tt a} might
actually be some section of an array.  In this example the blocking is
$(13,12,13,12)$ and we must take into account that the subscripting into
the local segment of the array is strided.  Also there is an offset of the
first element, which in some processes is zero and in others is one.

\begin{figure}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
                  Range x = new BlockRange(50, d) ;
                  float [[-]] a = new float [[x]] ;

\end{verbatim}
\end{code}
\end{displaymath}
%\psfig{figure=mapBlock.eps,width=4.9in}
\includegraphics[width=4.9in]{mapBlock.eps}
\begin{displaymath}
\begin{code}
\begin{verbatim}


                  Range x = new CyclicRange(50, d) ;
                  float [[-]] a = new float [[x]] ;

\end{verbatim}
\end{code}
\end{displaymath}
%\psfig{figure=mapCyclic.eps,width=4.9in}
\includegraphics[width=4.9in]{mapCyclic.eps}
\begin{displaymath}
\begin{code}
\begin{verbatim}


                  Range x = new BlockRange(100, d) ;
                  float [[-]] b = new float [[x]] ;

                  float [[-]] a = b [[0 : 99 : 2]] ;

\end{verbatim}
\end{code}
\end{displaymath}
%\psfig{figure=mapStride.eps,width=4.9in}
\includegraphics[width=4.9in]{mapStride.eps}
\begin{verbatim}

\end{verbatim}

\caption{Example embeddings of array elements in local
blocks\label{fig:mapExamples}}
\end{figure}

At this point one might feel inclined to abandon the idea of dimension
splitting.  These examples doesn't seem to fit at all with the idea of
dividing a distributed dimension of extent {\tt N} into {\tt P} blocks
of constant size {\tt B}, which is what is needed if dimension splitting
is to work.

Persistence will pay off.  For the sake of making progress, we note
that for a given range there is a fixed {\em bound} on the number
of array elements held by any process.  We can now redefine the symbol
{\tt B} to refer to this constant bound.  The operational assumption is
that {\em all} processes allocate enough space to hold this bounding
number of elements, although not all processes necessarily use all slots.
For the dimension-split array, the extent of the the sequential dimension
is the constant number, {\tt B}, of locally allocated slots.  The elements
of the original array are embedded somehow in these slots.

In Figure \ref{fig:mapExamples} the most likely value
for {\tt B} in the first two examples is $\lceil 50 / 4 \rceil = 13$.
For the third example the amount of space allocated will presumably
be enough to hold the parent distributed array---most likely {\tt B} is
$\lceil 100 / 4 \rceil = 25$%
\footnote{In this case the dimension-split version of
{\tt a} is the same as the dimension-split version of the parent array
{\tt b}.  A curious feature of a section with a dimension-splitting
subscript is that the ``section'' may have {\em more} accessible
elements than the parent array.  This is a little counter-intuitive,
and introduces some possibilities for abuse.  But it is not a logically
inconsistent situation, and, used carefully, it has various benefits.}%
.

The embedding of actual elements is determined by a new
method, {\tt local\-Block()}.  This is a member of the {\tt Range} class.
It has no arguments, and returns
an object of class {\tt Block}, declared as:
\begin{displaymath}
\begin{code}
\begin{verbatim}
  class Block {
    public int count ;

    public int sub_bas ;
    public int sub_stp ;

    public int glb_bas ;
    public int glb_stp ;
  }
\end{verbatim}
\end{code}
\end{displaymath}
The value of {\tt count} specifies the number of actual array elements
in the selected block, and the pair {\tt sub\_bas}, {\tt sub\_stp}
define a base and step for the local subscript values associated with
those elements.  The pair {\tt glb\_bas}, {\tt glb\_stp} define a base
and step for the {\em global} index values associated with the elements.

(Note that unfortunately there is a overlap of nomenclature between
the blocks of an arbitrary range, and the distribution format of one
particular kind of range: {\tt BlockRange}.  The {\tt Block} class and
the {\tt localBlock()} method are in no way specifically tied to the
``block-wise'' distribution format embodied in {\tt BlockRange}.  Equivalent
methods are defined for any range.)

One of the most important applications of the {\tt localBlock()} method is
in a translation scheme for the {\tt overall} construct, and this is a natural
way to illustrate its use.

Consider this fragment of HPJava:
\begin{displaymath}
\begin{code}
\begin{verbatim}
  float [[-]] a;

  Range x = a.rng(0);

  overall(i = x for :)
    a [i] = (float) i` ;
\end{verbatim}
\end{code}
\end{displaymath}
By applying dimension splitting to the array {\tt a}, the {\tt overall}
construct can be translated as illustrated in Figure \ref{fig:recursiveOverall}.

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
\begin{code}
\begin{verbatim}
      float [[-]] a;

      Range x = a.rng(0);

      overall(i = x for :)
        a [i] = (float) i` ;
\end{verbatim}
\end{code}
\end{displaymath}
\textbf{TRANSLATION:}
\begin{displaymath}
\begin{code}
\begin{verbatim}
      float [[-]] a;

      Range x = a.rng(0);

      float [[-,*]] as = a [[<>]] ;

      Range d = as.rng(0) ;

      overall(k = d for :) {
        Block b = x.localBlock();
    
        for (int l = 0 ; l < b.count ; l++)
          as [k, b.sub_bas + b.sub_stp * l] = 
                              (float) (b.glb_bas + b.glb_stp * l) ;
      }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Recursive translation of a simple {\tt overall} construct.}
\label{fig:recursiveOverall}
\end{figure}

At first sight we have just replaced one {\tt overall} construct with another.
But recall that the range {\tt d} can be assumed to be a process dimension,
so the role of the new {\tt overall} construct is essentially
``formal''---{\tt k} only has one location in each process, so the new
{\tt overall} yields no local loop.  Although the {\tt k} subscript
of {\tt as} is required by the rules of the language, it is essentially
does nothing, again because there is only one location.  In fact
the only substantive effect of the reduced construct is to change the
active process group inside its body.

Effectively, this transformation has reduced the {\tt overall} construct
to a sequential local {\em for} loop.  Subscripting with a distributed
index has essentially been reduced to subscripting into the sequential
local array {\tt as[[k, :]]}.  Moreover, the subscript expression and
the global index expression on the right hand side have been reduced to
expressions linear in the loop index.  Such expressions can be translated
efficiently by a compiler using a {\em strength reduction} optimization
(replacing the linear expressions with an incremented accumulation
variable).

Because it involves reducing an {\tt overall} construct to a kind of
lower-level {\tt overall}, we call this general scheme {\em
recursive translation}.
%  The {\tt localBlock()} inquiry has
%variants that directly support triplet arguments to the {\tt overall}
%index range, but we defer discussion to section
%\ref{sec:subrangeBlocks}.

Note that technically the {\tt localBlock()} inquiry is {\em incoherent}
(see section \ref{sec:canonical}).  This could be fixed by requiring
an argument that specified the local coordinate, similar to the argument
of the coherent (though otherwise lower-level) {\tt block()} method that
will be introduced in section \ref{sec:coordRanges}.  This argument
was omitted here to simplify usage, and also to allow the unique result of {\tt
localBlock()} to be computed once and cached inside the {\tt Range}
object.  Effectively the incoherent {\tt crd()} inquiry is used
internally.

\subsection{Ghost regions and dimension splitting}
\label{sec:ghostSplitting}

If the distributed range has ghost extensions, this does {\em not} affect
the values in the block description returned by {\tt localBlock()}.  
These values describe the layout of elements associated with
the ``physical'' portion of the array, not elements in the ghost
region.
In this case, however, the range of legal subscripts in the local
sequential array dimension
is increased.  In the absence of ghost regions that range may be, for example,
$0, \ldots, B - 1$, where typically $B$ would be $\lceil N / P \rceil$.
If the original range has lower and upper ghost extensions of width 
$w_{\mbox{\small lo}}, w_{\mbox{\small hi}}$, so does the
new sequential range, exposed by dimension splitting.  The allowed range
of local subscripts will be 
$-w_{\mbox{\small lo}}, \ldots, B + w_{\mbox{\small hi}} - 1$%
\footnote{Note however that the {\tt size()} inquiry applied to the
associated collapsed range will still return the ``physical'' extent, $B$.}%
.
Figure \ref{fig:recursiveShifted} gives the recursive translation
of an {\tt overall} construct involving a shifted index.  It assumes
{\tt a} has a suitable ghost extensions.%
\footnote{This example actually assumes $x$ has alignment stride of 1.
In general the displacement in the translation should be multiplied by
$x\mbox{\tt .str()}$, where $x$ is the range parametrizing the original
{\tt overall}.}%
.

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
\begin{code}
\begin{verbatim}
      float [[-]] a;

      Range x = a.rng(0);

      overall(i = x for :)
        a [i + 1] = ... ;
\end{verbatim}
\end{code}
\end{displaymath}
\textbf{TRANSLATION:}
\begin{displaymath}
\begin{code}
\begin{verbatim}
      float [[-]] a;

      Range x = a.rng(0);

      float [[-,*]] as = a [[<>]] ;

      Range d = as.rng(0) ;

      overall(k = d for :) {
        Block b = x.localBlock();
    
        for (int l = 0 ; l < b.count ; l++)
          as [k, b.sub_bas + b.sub_stp * l + 1] = ... ;
      }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Recursive translation of a shifted index subscript.}
\label{fig:recursiveShifted}
\end{figure}

%While not an idiom to be highly recommended, it is {\em possible}
%to use dimension splitting to pull out the ghost regions of an array,
%and expose them as if they were normal array sections.  In:
%\begin{displaymath}
%\begin{code}
%\begin{verbatim}
%  Procs2 p = new Procs2(P, P) ;
%  on (p) {
%    Range x = new ExtBlockRange(N, p.dim(0), 1, 1) ;
%    Range y = new ExtBlockRange(N, p.dim(1), 1, 1) ;
%
%    float [[-,-]] a = new float [[x, y]] ;
%
%    float [[-,-,*]] as = a [[:, <>]] ;
%
%    float [[-]] glo = as [[:, 0, -1]] ;  // Negative block subscript
%    ...
%  }
%\end{verbatim}
%\end{code}
%\end{displaymath}
%the section {\tt glo} is a one dimensional array with range {\tt x},
%holding the ghost region of {\tt a} associated with the extreme lower
%edge of the {\tt y} dimension.  The situation is illustrated in
%Figure \ref{fig:ghostSection}.  One {\em could}, perhaps, use {\tt glo}
%to hold boundary values in a program solving a partial differential
%equation.  In practise it is almost certainly better to increase the
%``physical'' extent of the array to hold such boundary values.
%
%\begin{figure*}[tbp]
%\centerline{\psfig{figure=ghostSection.eps,width=4.5in}}
%\caption{Extreme lower ghost region exposed as a section (shaded
%area).\label{fig:ghostSection}}
%\end{figure*}

\subsection{Local blocks of subranges}
%\label{sec:subrangeBlocks}

So far our recursive translation scheme does not apply to general {\tt
overall} constructions, which include some non-default triplet parameters.
Overloaded versions of {\tt localBlock()} that take {\tt l}, {\tt u},
{\tt s} arguments are provided.  Translation in this case is illustrated
in Figure \ref{fig:tripletRecursiveOverall}.  There is also a version
of {\tt localBlock()} that omits the stride argument, {\tt s}.  This can
be used in the case of unit stride.

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
\begin{code}
\begin{verbatim}
      float [[-]] a;

      Range x = a.rng(0);

      overall(i = x for l : u : s)
        a [i] = (float) i` ;
\end{verbatim}
\end{code}
\end{displaymath}
\textbf{TRANSLATION:}
\begin{displaymath}
\begin{code}
\begin{verbatim}
      float [[-]] a;

      Range x = a.rng(0);

      float [[-,*]] as = a [[<>]] ;

      Range d = as.rng(0) ;

      overall(k = d for :) {
        Block b = x.localBlock(l, u, s);
    
        for (int l = 0 ; l < b.count ; l++)
          as [k, b.sub_bas + b.sub_stp * l] = 
                              (float) (b.glb_bas + b.glb_stp * l) ;
      }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Recursive translation of {\tt overall} construct with triplet index range.}
\label{fig:tripletRecursiveOverall}
\end{figure}

\clearpage

\section{Reduction to Java arrays}

Dimension splitting allows one to access the local blocks of distributed
arrays as sequential HPJava arrays.  In many cases this may be all one
needs to do low level SPMD programming.  But the translation scheme
for HPJava actually assumes that ultimately all Fortran-like arrays
are implemented in terms of the standard arrays of Java.  The spirit of
HPspmd languages is not to conceal such things.  In fact the spirit of
HPspmd languages is to shamelessly expose internal workings at all levels.
Hence the underlying Java arrays should be available if they are needed.

The inquiry {\tt dat()} can be applied to any HPJava array.  It returns
a reference to a Java array with the same type of elements as the target
array.  This is the actual array in which the local elements are stored.

This gives us yet another way to optimize the original data-parallel
N-body example of Figure \ref{fig:data-Nbody}.  We can express the
compute loop in the MPI style of Figure \ref{fig:mpi-Nbody}.
The code is given in Figure \ref{fig:reduced-Nbody}.
This particular implementation assumes that the process grid {\tt p}
coincides with the MPI group associated with the {\tt COMM\_WORLD}
communicator.

\begin{figure}[t]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs1 p = new Procs1(P) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;

    float [[-]] f = new float [[x]], a = new float [[x]],
                b = new float [[x]] ;

    ... initialize `a' ...

    overall(i = x for :) {
      f [i] = 0.0 ;
      b [i] = a [i] ;
    }

    // extract the local vectors of elements:
 
    float [] f_blk = f.dat(), a_blk = a.dat(), b_blk = b.dat() ;

    int myID = MPI.COMM_WORLD.Rank();

    for(int s = 0 ; s < P ; s++) {

      for(int i = 0 ; i < B ; i++)  // B : local block size
        for(int j = 0 ; j < B ; j++)
          f_blk [i] += force(a_blk [i], b_blk [j]) ;


      // cyclically shift `b_blk'...

      int right = (myID + 1) % P, left = (myID + P - 1) % P ;

      MPI.COMM_WORLD.Sendrecv_replace(b_blk, 0, B, MPI.FLOAT,
                                      right, 0, left, 0) ;
    }
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Version of the N-body force computation using reduction to
Java arrays\label{fig:reduced-Nbody}}
\end{figure}

Again this simple example hides the complexities that arise if we have
to deal with a general distributed array.  As we saw in the previous
section, the local elements of a general distributed array are effectively
stored in a multidimensional sequential array.  The detailed embedding is
defined by the {\tt localBlock()} inquiry on the distributed array ranges.

The mapping of the local sequential multidimension array into the Java
array is defined in turn by new inquiries {\tt bas()} and {\tt str()}
on an HPJava array.  These integer-valued methods define a base offset,
and a stride for each dimension%
\footnote{Note these ``memory'' bases and strides are distinct from
the subrange alignment parameters returned
by similarly named inquiries on ranges.}%
.
If an element of a sequential HPJava array,
{\tt a}, has integer subscripts $i_0, \ldots, i_{R-1}$
it is stored in element
\begin{displaymath}
\mbox{\tt a.dat() [a.bas()} + i_0 \times \mbox{\tt a.str(0)} + \ldots +
                              i_{R - 1} \times \mbox{\tt a.str(}R - 1\mbox{\tt )]}
\end{displaymath}
of the local Java array.

This formula can be extended to a formula for finding the local elements
of distributed arrays.  First we note that through dimension splitting
any distributed array can be reduced to an array that has a mix of
only sequential dimensions and level 0 ranges.  The level 0 ranges contribute
nothing to the total offset of the element in the local Java array.
So if the subscript list is $i_0, \ldots, i_{R-1}$---a mix of
integers and level 0 distributed index symbols---the local element is
\begin{displaymath}
\mbox{\tt a.dat() [a.bas()} \quad + 
    \sum_{\begin{array}{c} r, \\
             \parbox{0.8in}{\footnotesize \centering
                            $r$th subscript \\ an integer}
               \end{array}}
             i_r \times \mbox{\tt a.str($r$)}
\mbox{\tt ]}
\end{displaymath}
To complete the story, we need to know how the values
of the {\tt str()} and {\tt bas()} members are defined on a dimension
split array.  If {\tt as} is a section defined by splitting dimension
$r$ of an array {\tt a}, to yield dimensions $r'$ and $r'+1$ of {\tt as}, then 
\begin{itemize}
\item
{\tt as.bas()} is equal to {\tt a.bas()},
\item
{\tt as.str($r'+ 1$)} (the stride associated with the
new sequential dimension) is equal to {\tt a.str($r$)}, and
\item
{\tt as.str($r'$)} (the stride associated with the new distributed
dimension) is equal to $\mbox{\tt a.str($r$)}
\times \mbox{\tt as.rng($r' + 1$).volume()}$.  
\end{itemize}
The complicated final definition only really matters in the block
cyclic case (section \ref{sec:blockCyclic}), where we have to apply
dimension splitting a second time.  Temporarily ignoring that case, we can
use our new formulae to further simplify the recursive translation given
earlier.  The new translation is given in Figure \ref{fig:localOverall}.

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
\begin{code}
\begin{verbatim}
      float [[-]] a;

      Range x = a.rng(0);

      overall(i = x for :)
        a [i] = (float) i` ;
\end{verbatim}
\end{code}
\end{displaymath}
\textbf{TRANSLATION:}
\begin{displaymath}
\begin{code}
\begin{verbatim}
      float [[-]] a ;

      Range x = a.rng(0) ;

      Block b = x.localBlock() ;
    
      for (int l = 0 ; l < b.count ; l++)
        a.dat() [a.bas() + (b.sub_bas + b.sub_stp * l) * a.str(0)] = 
                            (float) (b.glb_bas + b.glb_stp * l) ;
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Reduction of subscripting in simple {\tt overall} construct to local
Java array accesses.}
\label{fig:localOverall}
\end{figure}

\clearpage

\section{Local arrays}

We have seen that the HPJava language provides two complementary ways
to access the local part of a distributed array---through the dimension
splitting syntax, and through the {\tt dat()} inquiry.

Neither of these return exactly what one might originally have
expected---a local sequental array containing exactly the local
elements of the distributed array---no more and no less.

We refrained from complicating the language
definition with this functionality, because it can
now be implemented using a library function.  For example,
for a two dimensional array of {\tt float}
the following procedure would do the job%
\footnote{This doesn't work if either array dimension has block-cyclic
distribution}%
:
\begin{displaymath}
\begin{code}
\begin{verbatim}
  public static float [[*,*]] local(float [[-,-]] a) {
    Range x = a.rng(0), y = a.rng(1) ;

    Block b = x.localBlock(), c = y.localBlock() ;

    int b_sub_top = b.sub_bas + (b.count - 1) * b.sub_stp ;
    int c_sub_top = c.sub_bas + (c.count - 1) * c.sub_stp ;

    float [[-,*,-,*]] as = a [[<>, <>]] ;

    return as [[x.dim().crd(), b.sub_bas : b_sub_top : b.sub_stp,
                y.dim().crd(), c.sub_bas : c_sub_top : c.sub_stp]] ; 
  }
\end{verbatim}
\end{code}
\end{displaymath}

\clearpage

\section{An extended example: prefix computation}
\label{sec:prefixComputation}

In this section we will give a non-trivial example of how dimension
splitting can be combined with the new inquiries on the {\tt Range}
class to write optimized parallel code that works for input arrays with
general distribution formats.

A prefix computation (also sometimes known as the ``scan'' operation)
takes an array as input, and outputs an array containing the
set of partial sums of the elements of that array.  If the input
array is {\tt a} and the result array is {\tt r}, we want the
outcome to be:
\begin{eqnarray*}
  \mbox{\tt r [} i \mbox{\tt ]} & = & 
  \mbox{\tt a [} 0 \mbox{\tt ]} + \ldots + \mbox{\tt a [} i \mbox{\tt ]}
\end{eqnarray*}
This is the {\em inclusive} form of prefix computation.  There is
also an {\em exclusive} form which would be defined by
\begin{eqnarray*}
  \mbox{\tt r [} 0 \mbox{\tt ]} & = &  0 \\
  \mbox{\tt r [} i \mbox{\tt ]} & = & 
  \mbox{\tt a [} 0 \mbox{\tt ]} + \ldots + \mbox{\tt a [} i - 1 \mbox{\tt ]}
      ; \quad 1 \le i < N
\end{eqnarray*}
We will be interested in computing the inclusive form, but sometimes
the exclusive form is needed in intermediate steps.

We can give a straightforward data-parallel algorithm for this
computation using a {\em doubling} technique.  Possible code is given
in Figure \ref{fig:doublingPrefix}.  The algorithm is very simple.
In a given iteration, the current value in the result array is shifted
by an amount that doubles between iterations.  The shifted array is
added into the current array.  The algorithm takes $\log_2(N)$
iterations to complete.  Its operation is illustrated in Figure
\ref{fig:doublingPrefix2}.

\begin{figure}[btp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
                void doublingPrefix(float [[-]] a) {
                  Group p = a.grp() ;

                  Range x = a.rng(0) ;
                  int N = x.size();

                  on(p) {
                    float [[-]] t = new float [[x]] ;

                    for(int s = 1 ; s < N ; s *= 2) {
                      Adlib.shift(t, a, s) ;
        
                      overall(i = x for s : N - 1)
                        a [i] += t [i] ;
                    }
                  }
                }

\end{verbatim}
\end{code}
\end{displaymath}
\caption{Simple doubling algorithm for parallel
prefix\label{fig:doublingPrefix}}
\end{figure}

\begin{figure}[btp]
%\centerline{\psfig{figure=doublingPrefix.eps,width=5in}}
\centerline{\includegraphics[width=5in]{doublingPrefix.eps}}

\caption{Illustration of doubling algorithm for parallel
prefix for N = 7.\label{fig:doublingPrefix2}}
\end{figure}

As often happens, the pure data parallel program is concise and quite
readable.  However it isn't very efficient.  It requires a total of
$N \times \log_2(N)$ floating point additions, whereas the naive sequential
algorithm only needs $N$.  So we can only expect useful parallel speedup
if $P \gg \log_2(N)$.  The pure data parallel version
needs optimization.

\subsection{Optimization for block distribution formats\label{sec:blockPrefix}}

For the most straightforward, block-based distribution formats---these
include arrays parameterized {\tt BlockRange}, {\tt ExtBlockRange}
and {\tt IrregRange}, there is a fairly straightforward optimization.
We can do prefix computation within individual blocks, then do a global
exclusive prefix combining the sums of the blocks.  Finally we add the
global prefix back to the incomplete prefixes within the blocks.  This is
illustrated in Figures \ref{fig:blockPrefix}, \ref{fig:blockPrefix2}.

\begin{figure}[btp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
      void blockPrefix(float [[-]] a) {
        Range x = a.rng(0) ;

        float [[-,*]] as = a [[<>]] ;

        Range z = as.rng(0) ;

        float [[-]] t = new float [[z]], s = new float [[z]] ;

        // 1. Do intra-block prefix.  Set `t' elements to sums of blocks

        overall (k = z for :) {
          Block b = x.localBlock() ;

          float sum = 0.0f ;
          for (int l = 0 ; l < b.count ; l++) {
            int sub = b.sub_bas + b.sub_stp * l ;

            sum = (as [k, sub] += sum) ;
          }

          t [k] = s [k] = sum ;
        }

        // 2. Do "global prefix"

        prefix(t) ;  

        // 3. Add exclusive global prefix to intra-block prefixes in `a'

        overall (k = z for :) {
          Block b = x.localBlock() ;

          float sum = t [k] - s [k] ;

          for (int l = 0 ; l < b.count ; l++) {
            int sub = b.sub_bas + b.sub_stp * l ;

            as [k, sub] += sum ;
          }
        }
      }
\end{verbatim}
\end{code}
\end{displaymath}

\caption{Parallel prefix optimized for block-wise distribution
formats.\label{fig:blockPrefix}}
\end{figure}

\begin{figure}[btp]
%\centerline{\psfig{figure=blockPrefix.eps,width=5in}}
\centerline{\includegraphics[width=5in]{blockPrefix.eps}}
\caption{Prefix optimized for block distributions: illustration
for N = 8, P = 3.\label{fig:blockPrefix2}}
\end{figure}

This version has a hope to be reasonably efficient if $N \gg P$, so
arithmetic costs have a chance to dominate communication cost.  It still
does about $2N$ total additions operations instead the $N$ operations for
the sequential code.  But in principle if $P > 2$ it should be possible
to compensate for this factor, and gain some parallel speedup.

The code given here will work for the ranges mentioned above,
and subranges of these%
\footnote{At least for subranges with positive alignment stride.  We will
return to this issue in section \ref{sec:generalPrefix}}%
.  It will not work for cyclic distributions.

\subsection{Cyclic distribution formats}

It seems difficult to give a truly efficient parallel prefix algorithm
when the data has cyclic distribution format.  It looks like an {\em IO
bound} problem.  It is certainly possible to find optimized algorithms
that are {\em better} than our naive data parallel version---for example
reducing the number of communication operations.  

In general, even if an operation like this cannot be implemented with
really good arithmetic performance, it does not follow that it is a
useless operation, or that it is not worth optimizing.  The operation in
question might be a necessary step in larger program (for example the
block-cyclic prefix that we will mention in section \ref{sec:blockCyclic}).
So it may be worth our while to optimize the procedure so that it does
not form a bottleneck in the larger context, even if we can't make it
truly efficient as an arithmetic operation in its own right.

An improved scheme is illustrated in Figure \ref{fig:cyclicPrefix2}.
To make the example a bit more interesting, it covers the case of an
array with a non-trivial (stride 2) alignment to a cyclic range.

%\begin{figure}[btp]
%\begin{displaymath}
%\begin{code}
%\begin{verbatim}
%      void cyclicPrefix(float [[-]] a) {
%        Range x = a.rng(0) ;
%
%        int [[-]] t = new int [[x]] ;
%
%        // 1. Copy `a' elements `t'.
%
%        overall(i = x for :)
%          t [i] = a [i] ;
%
%        int [[-,*]] ts = t [[<>]] ;   // Stages 2-4 process `t' by blocks
%
%        Range z = ts.rng(0) ;
%        int   B = ts.rng(1).size() ;
%
%        // 2. Global prefix across blocks
%
%        vecPrefix(ts) ;
%
%        // 3. Broadcast last column of the blocks prefix.
%
%        int [[*]] sums = new int [[B]] ;
%
%        Adlib.remap(sums, ts [[z.size() - 1, :]]) ;
%
%        // 4. Add exclusive prefix of `sums' to local `ts'
%
%        overall (k = z for :) {
%
%          int total = 0.0 ;
%          for(int l = 0 ; l < B ; l++) {
%            ts [k, l] += total ;
%            total += sums [l] ;
%          }
%        }
%
%        // 5. Copy relevant elements of `t' back to `a'
%
%        overall(i = x for :)
%          a [i] = t [i] ;
%      }
%    }
%\end{verbatim}
%\end{code}
%\end{displaymath}
%
%\caption{Parallel prefix optimized for cyclic distribution
%format.  (See discussion in section \ref{sec:generalCyclic}
%for modifications to handle negative alignment strides.)
%\label{fig:cyclicPrefix}}
%\end{figure}

\begin{figure}[btp]
%\centerline{\psfig{figure=cyclicPrefix.eps,width=5in}}
\centerline{\includegraphics[width=5in]{cyclicPrefix.eps}}
\caption{Possible optimization of prefix for cyclic distributions: illustration
for stride 2 subrange of cyclic range with extent 15.
$\mbox{\tt P} = 5$.\label{fig:cyclicPrefix2}}
\end{figure}

The input array is copied to a temporary (dimension split) array
{\tt ts}, with zeros in positions for which there is no corresponding
element of {\tt a}.  Then the global prefixes are formed across individual
rows (as drawn here).  With the the blocks (columns) treated as vectors,
we would only need $\log_2(\mbox{\tt P})$ shift-type operations to do this,
if we used a naive doubling algorithm for this stage.

%A simple implementation of {\tt vecPrefix} is given in
%Figure \ref{fig:vecPrefix}.
%
%\begin{figure}[btp]
%\begin{displaymath}
%\begin{code}
%\begin{verbatim}
%                void vecPrefix(float [[-,*]] a) {
%                  Range x = a.rng(0) ;
%                  int N = x.size();
%
%                  int B = a.rng(1).size();
%
%                  float [[-,*]] t = new float [[x, B]] ;
%
%                  for(int s = 1 ; s < N ; s *= 2) {
%                    Adlib.shift(t, a, s, 0) ;
%        
%                    overall (i = x for s : N - 1)
%                      for (int j = 0 ; j < B ; j++)
%                        a [i, j] += t [i, j] ;
%                  }
%                }
%
%\end{verbatim}
%\end{code}
%\end{displaymath}
%\caption{Simple implementation of vector prefix\label{fig:vecPrefix}}
%\end{figure}

An exclusive prefix of the final column---the row sums---is broadcast
to all processes.  This exclusive prefix is added to the incomplete
prefixes across the rows, computed previously.  The results are copied
back to {\tt a}.

%We have relied on the fact that Java initializes new arrays to zero.
%If you prefer not to rely on this feature, you could initialize the whole
%of {\tt ts} explicitly using
%\begin{displaymath}
%\begin{code}
%\begin{verbatim}
%        HPspmd.init(ts, 0.0) ;
%\end{verbatim}
%\end{code}
%\end{displaymath}
%{\em before} the elements of {\tt a} are copied to {\tt t}.

It is straightforward enough to implement this scheme in HPJava using
dimension splitting.  However the improvement in efficiency is probably
not dramatic, and we will save space here by omitting the specialized
code.  To tell the truth the ``naive'' data parallel version is probably
not much worse in practise.

\subsection{Optimization for ``general'' distribution formats}
\label{sec:generalPrefix}

We can combine the procedures given in the preceding sections to produce
a single optimized {\tt prefix} procedure that works for any distribution
format by using the {\tt format()} inquiry on {\tt Range}.  The code
is given in Figure \ref{fig:generalPrefix}.

The inquiry {\tt format()} returns the constant {\tt DIST\_DIMENSION}
if the range is a process dimension and {\tt DIST\_CYCLIC} if it is
a cyclically distributed range (or subrange).  In these two cases
we use the naive algorithm.  In all other cases we use the improved
{\tt blockPrefix()}.

Notice that {\tt blockPrefix()} will work OK for a collapsed range
(corresponding to a sequential array).  It is permitted to do dimension
splitting on a collapsed array.  The range of the resulting
distributed dimension is a ``degenerate'' internal process dimension 
of size 1.  Everything will work, although it might be
more efficient to test for {\tt DIST\_COLLAPSED} and handle this
in a separate, optimized subroutine.
{\em [Yes.  Add the code to do this.]}

\begin{figure}[btp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
                void prefix(float [[-]] a) {
                  Group p = a.grp() ;

                  on(p) {
                     Range x = a.rng(0) ;

                     switch (x.format()) {

                       case Range.DIST_DIMENSION :
                       case Range.DIST_CYCLIC :

                         doublingPrefix(a) ;
                         break ;

                       default :
                         blockPrefix(a) ;
                     }
                   }
                 }

\end{verbatim}
\end{code}
\end{displaymath}
\caption{Optimized parallel prefix for any distribution format.
\label{fig:generalPrefix}}
\end{figure}

In the interests of simplifying the presentation, we left a bug in the
algorithm of section \ref{sec:blockPrefix}.  It will fail in the
case where the original array range is a subrange with {\em negative}
alignment stride.  {\em [Give an example.]}  
To deal with this case the ``recursive'' call to
compute global prefixes could be replaced with a call that computes
the global {\em suffix}---partial sums of elements that start at the
topmost element and increment downwards {\em [Give a formal definition]}.  
So stage 2 in
Figure \ref{fig:blockPrefix} could be replaced with something like%
\footnote{The inquiry {\tt str()} on {\tt Range} returns the alignment
stride of an arbitrary range.  There is a related inquiry, {\tt bas()},
that returns the alignment base.  These alignment parameters
of ranges are distinct from ``memory'' base and strides returned
by the similarly-named inquiries on distributed arrays.}%
:
\begin{displaymath}
\begin{code}
\begin{verbatim}
        if(x.str() > 0)
          prefix(t) ;
        else
          suffix(t) ;
\end{verbatim}
\end{code}
\end{displaymath}
The suffix computation procedure can use essentially the same
algorithms as the prefixes, with some loop orders and shift
directions reversed.  A slightly more elegant solution to this problem
will be given in section \ref{sec:crds}.
Note that there is no need to change the individual block processing
to take account of negative alignment strides: the {\tt localBlock()} inquiry
does this automatically.  The {\tt sub\_stp} field will be negative in
this case.

There is one distribution format we have not considered here---{\em block
cyclic} distribution format.  Section \ref{sec:blockCyclic} will explain 
one way HPJava can deal with this case.

\clearpage

\section{Non-local blocks}
\label{sec:coordRanges}

Section \ref{sec:blockParameters} described the relatively easy-to-use {\tt
localBlock()} inquiry.  This inquiry returned block parameters for the
local coordinate value in the process dimension associated with the range.

There are other situations, for example inside the implementation of
communication functions like {\tt Adlib.remap}, where {\em all}
blocks---local {\em and} remote---of a subrange must be enumerated.

To deal with these situations there is a general method called {\tt
block()}.  This method takes one integer argument---a coordinate in the
process dimension associated with the range.  Like {\tt localBlock()}
it returns a {\tt Block} object, defining the layout of
distributed array elements in that process.

One might expect the {\tt block()} inquiry should be well-defined for any
valid process coordinate.  However there are some array alignment options
that such a scheme doesn't handle particularly well.  Consider
the example array sections illustrated in Figure \ref{fig:mapSubExamples},
which presume the declarations:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    Procs p = new Procs1(6) ;
    Dimension d = p.dim(0) ;
\end{verbatim}
\end{code}
\end{displaymath}

\begin{figure}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
                  Range x = new BlockRange(20, d) ;
                  float [[-]] b = new float [[x]] ;

                  float [[-]] a = b [[5 : 10]] ;

\end{verbatim}
\end{code}
\end{displaymath}
%\psfig{figure=mapSubBlock.eps,width=4.9in}
\includegraphics[width=4.9in]{mapSubBlock.eps}
\begin{displaymath}
\begin{code}
\begin{verbatim}


                  Range x = new CyclicRange(20, d) ;
                  float [[-]] c = new float [[x]] ;

                  float [[-]] a = c [[0 : 18 : 2]] ;

\end{verbatim}
\end{code}
\end{displaymath}
%\psfig{figure=mapSubCyclic.eps,width=4.9in}
\includegraphics[width=4.9in]{mapSubCyclic.eps}
\begin{displaymath}
\begin{code}
\begin{verbatim}


                  Range x = new BlockRange(20, d) ;
                  float [[-]] b = new float [[x]] ;

                  float [[-]] a = b [[19 : 0 : -1]] ;

\end{verbatim}
\end{code}
\end{displaymath}
%\psfig{figure=mapSubNegative.eps,width=4.9in}
\includegraphics[width=4.9in]{mapSubNegative.eps}
\begin{verbatim}

\end{verbatim}

\caption{Example sections with unusual coordinate ranges.}
\label{fig:mapSubExamples}
\end{figure}

The first example involves a narrow subrange of a block distributed range.
Only two of the six processes hold any elements of the array section.
This isn't a problem with for the simple version of block parametrization
presented in section \ref{sec:blockParameters}---the {\tt localBlock()}
inquiry will simply return ``empty'' blocks, with {\tt count} set to zero.
When translating simple {\tt overall} constructs, the overhead of
calling {\tt localBlock()} unconditionally in all processes is not
a problem---the method is called at most once in each process anyway.
But in situations where {\em all} blocks---local {\em and} remote---of a
subrange must be enumerated it may become genuinely inefficient to blindly
work through every coordinate value, sifting through many empty blocks.

The second example in Figure \ref{fig:mapSubExamples}, illustrates that
a similar problem can arise for the case of a strided section of a {\em
cyclic} range.  In this particular example half the processes hold no
elements.  Again, blindly computing {\tt localBlock()} for all coordinates
can lead to inefficiencies, especially in communication functions
(where this kind of situation actually arises quite naturally when
one is dealing with cyclic ranges).

The final example of Figure \ref{fig:mapSubExamples} is rather different.
It illustrates the case of an array section with a {\em negative}
alignment stride.  The natural enumeration order for coordinates of blocks
is reversed here, and there are some situations where failure to take
this into account can lead to wrong results.  In fact we already saw
one such example in section \ref{sec:prefixComputation}.

\subsection{The {\tt crds()} method}
\label{sec:crds}

To allow for these kinds of situation, an new method {\tt crds()}
is added to the {\tt Range} class.
It takes no arguments, and returns
an object of class {\tt Triplet}, declared as:
\begin{displaymath}
\begin{code}
\begin{verbatim}
  class Triplet {
    public int lo, hi ;
    public int stp ;

    public boolean inRange(int n) {...}
  }
\end{verbatim}
\end{code}
\end{displaymath}
The values of {\tt lo}, {\tt hi} and {\tt stp} define the parameters
of some strided interval:
\begin{displaymath}
  \mbox{\tt \small lo}, \quad
  \mbox{\tt \small lo} + \mbox{\tt \small stp}, \quad
  \mbox{\tt \small lo} + 2 \times \mbox{\tt \small stp}, \quad \ldots, \quad
  \mbox{\tt \small lo} + 
    \left(\left\lfloor \frac{\mbox{\tt \small hi} - \mbox{\tt \small lo}}
                            {\mbox{\tt \small stp}} 
          \right\rfloor + 1\right) \times \mbox{\tt \small stp}
\end{displaymath}
The method {\tt inRange()} returns {\tt true} if and only if its
argument is in this interval.

The {\tt Triplet} object returned by the {\tt crds()} method defines a
triplet range of coordinates for which the {\tt block()} method
is well-defined.  
The method {\tt localBlock()} introduced earlier can be defined in terms of the
more primitive {\tt block()} as follows:
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Block localBlock() {
    int crd = dim().crd();
    if(crds().inRange(crd))
      return block(crd) ;
    else
      return Block.EMPTY ;
  }
\end{verbatim}
\end{code}
\end{displaymath}
The {\tt EMPTY} block can be assumed to have fields:
\begin{displaymath}
\begin{code}
\begin{verbatim}
  EMPTY.count   = 0 ;

  EMPTY.sub_bas = 0 ;
  EMPTY.sub_stp = 1 ;

  EMPTY.glb_bas = 0 ;
  EMPTY.glb_stp = 1 ;
\end{verbatim}
\end{code}
\end{displaymath}
Calling {\tt block()} for an argument outside the range defined
by {\tt crds()} is an error%
%\footnote{It is an unchecked error and the results are unpredictable.
%Definitely it should not be assumed to return an empty block.
%The implementation of {\tt block()} can save a precious few cycles by
%not doing the check.  These low levels of the HPspmd
%runtime are unforgiving!}%
.

As an immediate application of the new
methods, consider the parallel prefix of Figure \ref{fig:blockPrefix}.
We noted at the end of section \ref{sec:generalPrefix} that this
code fails for the case of a negative alignment stride.
An ad hoc solution was given there.  A more elegant and systematic
approach would be to replace stage 2 with:
\begin{displaymath}
\begin{code}
\begin{verbatim}
        Triplet crds = x.crds() ;

        prefix(t [[crds.lo : crds.hi : crds.stp]]) ;
\end{verbatim}
\end{code}
\end{displaymath}
If the original range has negative alignment stride, then the value of
{\tt crds.stp} will be negative.  The recursive call to {\tt prefix}
on a negative alignment stride section of {\tt t} effectively implements the
suffix computation on {\tt t} itself%
\footnote{This change will also improve
performance of the recursive call if {\tt x} has many empty blocks, as
in the first two array sections of Figure \ref{fig:mapSubExamples}.}%
.

\subsection{Blocks of subranges}
%\label{sec:subrangeBlocks}

As in the case of {\tt localBlock}, there are overloaded versions of {\tt
crds()} and {\tt block()} that take {\tt l}, {\tt u}, {\tt s} arguments.
Their use will be illustrated in a translation scheme for
{\tt overall} constructs parametrized by block cyclic ranges in 
the next section.

\clearpage

\section{Block cyclic distribution---up a level}
\label{sec:blockCyclic}

High Performance Fortran included the {\em block cyclic} distribution
format, whereby a distributed array range is divided into contiguous
blocks of some fixed size, and these blocks are distributed over the
process dimension in a cyclically wrapped way.  There are various
implementation problems associated with this distribution format, and
it is not universally accepted that its benefits really justify the
added complexity.  But we will dedicate this section to describing how
block cyclic distribution can be incorporated into the HPJava
framework described so far.

An individual process now generally holds many primitive blocks for
a range.  A straightforward local translation of an {\tt overall} construct
may now involve a pair of nested for loops: an inner loop over a
single block as before, and an outer loop enumerating the list of blocks
allocated to the local process.

One idea for dealing with this is to leave the syntax for dimension
splitting unchanged.  {\em But}, when we split
the array dimension in Figure \ref{fig:tripletRecursiveOverall},
the distributed range, {\tt d}, associated with dimension-split array
is now no longer a process dimension (class {\tt Dimension}).  Instead it
is a range of class {\tt CyclicRange}.  We call this cyclic range
the {\em kernel} of the original block-cyclic range.

The {\tt localBlock()} inquiry is ill-defined on a block-cyclic range,
because there are usually many local blocks.
We must generalize the recursive translation schemes presented in
section \ref{sec:blockParameters} to use the {\tt crds()} and {\tt
block()} inquires (Figure \ref{fig:improvedRecursiveOverall}).
%Then we apply recursive translation a second time to reduce to
%an {\tt overall} construct involving a process dimension (Figure
%\ref{fig:recursiveBlockCyclic}).
In the current example, the ``coordinates'' returned by the {\tt
crds()} inquiry are ``virtual coordinates'': they are actually global
subscripts in the cyclic kernel range.  The {\tt overall} construct
in the transformed code is no longer a trivial single-pass local loop.
On the other hand, it can be reduced by applying the original recursive
translation tranformation to the generated {\tt overall}, as illustrated
in Figure \ref{fig:recursiveBlockCyclic}%
\footnote{Use of the original
recursive translation scheme using {\tt localBlock()}, rather than the
generalized scheme using {\tt crds()} and {\tt block()}, is not mandatory
for the second application.  But it is allowed, and is slightly more
convenient.}%
.



\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
\begin{code}
\begin{verbatim}
      float [[-]] a;

      Range x = a.rng(0);

      overall(i = x for l : u : s)
        a [i] = (float) i` ;
\end{verbatim}
\end{code}
\end{displaymath}
\textbf{TRANSLATION:}
\begin{displaymath}
\begin{code}
\begin{verbatim}
      float [[-]] a;

      Range x = a.rng(0);

      float [[-,*]] as = a [[<>]] ;

      Range d = as.rng(0) ;

      Triplet t = x.crds(l, u, s);

      overall(k = d for t.lo : t.hi : t.stp) {
        Block b = x.block(k`, l, u, s);
    
        for (int l = 0 ; l < b.count ; l++)
          as [k, b.sub_bas + b.sub_stp * l] = 
                              (float) (b.glb_bas + b.glb_stp * l) ;
      }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Recursive translation of an {\tt overall} construct, using
{\tt crds()} and {\tt block()}.}
\label{fig:improvedRecursiveOverall}
\end{figure}


\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
\begin{code}
\begin{verbatim}
      float [[-]] a;

      Range x = a.rng(0);

      float [[-,*]] as = a [[<>]] ;

      Range d = as.rng(0) ;

      Triplet t = x.crds(l, u, s);

      overall(k = d for t.lo : t.hi : t.stp) {
        Block b = x.block(k`, l, u, s);
    
        for (int l = 0 ; l < b.count ; l++)
          as [k, b.sub_bas + b.sub_stp * l] = 
                              (float) (b.glb_bas + b.glb_stp * l) ;
      }
\end{verbatim}
\end{code}
\end{displaymath}
\textbf{TRANSLATION:}
\begin{displaymath}
\begin{code}
\begin{verbatim}
      float [[-]] a;

      Range x = a.rng(0);

      float [[-,*]] as = a [[<>]] ;

      Range d = as.rng(0) ;

      Triplet t = x.crds(l, u, s);

      float [[-,*,*]] ass = as [[<>,:]] ;

      Range e = ass.rng(0) ;

      overall(m = e for :) {
        Block c = d.localBlock(t.lo, t.hi, t.stp) ;

        for(int n = 0 ; n < c.count ; n++) {
          Block b = x.block(c.glb_bas + c.glb_stp * n, l, u, s) ;
    
          for (int l = 0 ; l < b.count ; l++)
            ass [m, c.sub_bas + c.sub_stp * n, b.sub_bas + b.sub_stp * l] = 
                              (float) (b.glb_bas + b.glb_stp * l) ;
        }
      }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Applying recursive translation a second time, for an {\tt overall} 
construct parametrized by a block cyclic range.}
\label{fig:recursiveBlockCyclic}
\end{figure}

%This may become a little clearer if we explicitly use the knowledge
%that the kernel is a simple cyclic range, and consider the common
%case where the kernel subscript range defined by {\tt t} is just
%the whole of the kernel.  In that case the fields of
%the block description record, {\tt c}, take the values:
%\begin{displaymath}
%\begin{code}
%\begin{tabbing}
%\verb$count   = $ $\lceil (K - \mbox{\tt crd}) / P \rceil$ \\
%\\
%\verb$sub_bas = 0$ \\
%\verb$sub_stp = 1$ \\
%\\
%\verb$glb_bas = crd$ \\
%\verb$glb_stp = $$P$
%\end{tabbing}
%\end{code}
%\end{displaymath}
%where $K$ is the extent of the kernel, {\tt d}, and {\tt crd} is the
%local coordinate associated with {\tt e}.  The nested loops simplify to:
%\begin{displaymath}
%\begin{code}
%\begin{verbatim}
%  overall(j = e for :) {
%    int count = (d.size() - j` + P - 1) / P ;
%
%    for(int m = 0 ; m < count ; m++) {
%      Block b = x.block(j` + P * m) ;
%    
%      for (int l = 0 ; l < b.count ; l++) {
%        ass [k, m, b.sub_bas + b.sub_stp * l] = 
%                          (float) (b.glb_bas + b.glb_stp * l) ;
%      }
%    }
%  }
%\end{verbatim}
%\end{code}
%\end{displaymath}
%The loop on {\tt m} is a loop over the locally held blocks, whose
%parameters are determined by the call to {\tt x.block()} call.

The correctness of this translation in the case where the
original array had {\em negative-stride alignment} (to a block cyclic
range) relies on the fact that {\tt t.stp} is negative in this case.
Otherwise the locally held blocks would not be enumerated in the correct
order, and we would not strictly implement the semantics for {\tt overall}
defined way back in section \ref{sec:locations}---viz that if {\tt s}
is negative then
\begin{displaymath}
\begin{code}
\begin{verbatim}
    overall(i = x for l : u : s) {
      ...
    }
\end{verbatim}
\end{code}
\end{displaymath}
is equivalent in behaviour to
\begin{displaymath}
\begin{code}
\begin{verbatim}
    for(int n = l; n >= u ; n += s)
      at(i = x [n]) {
        ...
      }
\end{verbatim}
\end{code}
\end{displaymath}

The parallel prefix implementation of section \ref{sec:prefixComputation}
needs only minor modifications to allow for arguments with block cyclic
distribution.  The enumeration of local blocks should be changed to use
{\tt crds()} and {\tt block()} (Figure \ref{fig:blockCyclicPrefix}).
The blocks of a block-cyclically distributed array will now be handled
correctly by {\tt blockPrefix()}, and the function will correctly recurse
to the cyclic code to handle the kernel prefix%
\footnote{It is a slightly esoteric point, but it is arguable
that the reason we had to make any changes at all is because the original
version used the incoherent {\tt localBlock()} inquiry.  If we had stuck
to the canonical HPspmd style, we would have been forced to use the coherent
{\tt block()} inquiry from the start.}%
.

\begin{figure}[btp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
      void blockPrefix(float [[-]] a) {
        Range x = a.rng(0) ;

        float [[-,*]] as = a [[<>]] ;

        Range z = as.rng(0) ;

        float [[-]] t = new float [[z]], s = new float [[z]] ;

        Triplet crds = x.crds();

        // 1. Do intra-block prefix.  Set `t' elements to sums of blocks

        overall (k = z for crds.lo : crds.hi : crds.stp) {
          Block b = x.block(k`) ;

          float sum = 0.0 ;
          for (int l = 0 ; l < b.count ; l++) {
            int sub = b.sub_bas + b.sub_stp * l ;

            sum = (as [k, sub] += sum) ;
          }

          t [k] = s [k] = sum ;
        }

        // 2. Do "global prefix"

        prefix(t [[crds.lo : crds.hi : crds.stp]]) ;

        // 3. Add exclusive global prefix to intra-block prefixes in `a'

        overall (k = z for crds.lo : crds.hi : crds.stp) {
          Block b = x.block(k`) ;

          float sum = t [k] - s [k] ;

          for (int l = 0 ; l < b.count ; l++) {
            int sub = b.sub_bas + b.sub_stp * l ;

            as [k, sub] += sum ;
          }
        }
      }
\end{verbatim}
\end{code}
\end{displaymath}

\caption{Block-wise version of parallel prefix modified to allow
for block-cyclic distribution formats.\label{fig:blockCyclicPrefix}}
\end{figure}


We will sometimes refer to block cyclic ranges as ``level 2'' ranges.
Most other distribution formats are associated with ``level 1'' ranges.
Process dimensions are ``level 0'' ranges.

\clearpage

%\end{document}

\chapter{Translation scheme}

This chapter describes a basic translation scheme for HPJava.
The translation scheme is given in some detail; in effect it represents
the most formal and complete definition of the language itself.

\section{Preliminaries}

\subsection{On distributed array types}
\label{sec:arrayTypes}

The most general distributed array type signature is:
\begin{displaymath}
T \verb$ [[$ \mbox{\it attr}_0, \ldots, \mbox{\it attr}_{R-1} \verb$]] $ \mbox{\it bras}
\end{displaymath}
where $T$ is any Java type other than an array type, $R$ is the rank of
the distributed array, each term $\mbox{\it attr}_r$ either consists
of a single hyphen, {\tt -}, or a single asterisk, {\tt *}, the term
$\mbox{\it bras}$ is a string of zero or more bracket pairs, {\tt []}.
This signature represents the type of a distributed array whose elements
have Java type
\begin{displaymath}
T \verb$ $ \mbox{\it bras}
\end{displaymath}

In normal Java, arrays are not considered to be objects, but they
are considered to have a class---a class representing an array type.
We prefer to avoid making the statement ``distributed arrays have a
class''.  If we made this statement it would probably commit us to either:
\begin{enumerate}
\item[A.]
\label{item:extendClass}
extending the definition of class in the Java base language, or
\item[B.]
\label{item:createClasses}
creating genuine Java classes for each type of HPJava
array that might be needed.
\end{enumerate}
Class is such a fundamental concept in Java that option A looks
infeasibly hard: would people expect us, for example, to integrate
the complex runtime inquiries on HPJava distributed arrays into some
extended version of the Java reflection API?  Or into the Java Native
Interface, JNI?  Such fundamental extensions to the Java platform are
almost certainly impractical.

Option B has its own problems.  Presumably the associated class types
should capture the rather complicated system of array types we have
described for HPJava.  Because there is an infinite number of
array types, the associated classes would certainly have to be created
on demand by the translator.  Does the translator have to create class
files for these automatically generated classes?  If so how should these
files be managed?  Distributed array types have a rather complex,
multiple-inheritance-like lattice of subtype relations, illustrated in
Figure \ref{fig:arrayTypes}.  This kind of type-lattice {\em can} be
reproduced in Java by using interface types.  But then, when we generate
a new array class, we have to make sure all the interfaces it implements
directly and indirectly are also defined.

\begin{figure}[tbp]
%\centerline{\psfig{figure=arrayTypes.eps,width=4.5in}}
\centerline{\includegraphics[width=4.5in]{arrayTypes.eps}}
\caption{Part of the lattice of types for distributed arrays of {\tt float}}
\label{fig:arrayTypes}
\end{figure}

We will finesse these issues by saying that an HPJava distributed array
has an ``extended class''.  The extended class concept embraces ordinary
Java classes and HPJava distributed array types, as separate concepts.

The fact that a distributed array is not a member of any Java class
has a real impact on how a distributed array can be used.
For example, a distributed array {\em cannot} be an element of an
ordinary Java array, nor can a distributed array reference be stored
in a standard library class like {\tt Vector}, which expects an {\tt
Object}.  In practise this is not such a drastic limitation as it sounds,
because the programmer can always create wrapper classes for particular
types of distributed array.  For example suppose we need a ``stack''
of two-dimensional distributed arrays of floating point numbers.  We can
set this up as follows:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    class Level implements hpjava.lang.HPspmd {

      public Level(float [[-,-]] arr) {this.arr = arr ; }

      public float [[-,-]] arr ;
    }


    Range x, y ;

    Level [] stack = new Level [S] ;
    for (int l = 0 ; l < S ; l++)
      stack [l] = new Level(new float [[x, y]]) ;
\end{verbatim}
\end{code}
\end{displaymath}
So the fact that distributed arrays cannot be
treated as normal objects is usually a minor inconvenience, not a fundamental
limitation%
\footnote{Note that Fortran 90 also does not allow arrays of arrays
to be declared directly.  If they are needed they have to be simulated
by introducing a derived datatype wrapper, just as we introduced a class
here.}%
.  The interface {\tt hpjava.lang.HPspmd} will be discussed in the next
section.

%Another random implication of the fact that distributed arrays do
%not belong to the {\tt Object} class is that they do not have a
%{\tt toString()} method (so, for example, a distributed array cannot be
%printed out using {\tt println}).  This is probably for the best, since
%converting a distributed array to a string containing a representation of
%all its elements involves a fairly complicated pattern of communication
%(and, possibly, synchronization).  Certainly it would not sit well with
%the HPspmd mindset to have such communications embedded in the
%implementation an operation that looks like standard Java.
%{\em [Red herring, because {\tt toString()} on an ordinary Java array
%doesn't do this.]}

\subsection{HPspmd classes}

We will define a translation scheme from HPJava class definitions
to standard Java-language class definitions.  The existing HPJava
translator literally goes through these stages, generating Java source
as its output.  In the future a more advanced HPJava compiler might
directly generate Java byte code, or even machine code.  Nevertheless,
the early phases of compilation will probably apply transformations similar
to the ones described here.

In general only a subset of the classes in an HPJava program will
actually use the special syntax of distributed arrays and distributed
control constructs.  Many of the classes used will be written
in standard Java, or may be part of standard Java libraries.

Methods that {\em do} use HPJava syntax have some special properties.
Apart from the fact that they may take distributed array arguments, they
also assume that there is a well-defined active process group (APG) at
their point of invocation.  So in general it is problematic (although
not impossible) to invoke HPJava code from a piece of ordinary Java.
These problems reflect genuine limitations of the underlying SPMD
programming model---it is difficult to {\em directly} invoke distributed
parallel procedures from sequential code.

The HPJava translator tries to make a clear distinction between
code that may use HPJava syntax extensions and Java code that may not.
It introduces a special interface, {\tt hpjava.lang.HPspmd}, which must
be implemented by any class that uses the special syntax.

We will refer to a class that implements the {\tt hpjava.lang.HPspmd}
interface as an {\em HPspmd class}.  Any other class is a {\em non-HPspmd
class}.  Likewise, an interface that extends the {\tt hpjava.lang.HPspmd}
interface is an {\em HPspmd interface}, and any other interface is a
{\em non-HPspmd interface}.  The extended syntax of HPJava can only be
used in methods, constructors and fields declared in HPspmd classes
and interfaces.  To discourage invocation of HPspmd code from non-HPspmd
code, the HPJava translator imposes the following limitations:
\begin{enumerate}
\item
An HPspmd interface may not extend {\em any} non-HPspmd interface.
\item
An HPspmd class may not implement {\em any} non-HPspmd interface.
\item
An HPspmd class may not extend any non-HPspmd class, except for the
mandatory base class, {\tt Object}.
\item
An HPspmd class may not override
the non-final methods from the base class {\tt Object}.
\end{enumerate}
{\em [Don't remember now why a weaker restriction---allowing non-HPspmd
base classes, but not allowing HPspmd
classes to override methods from these bases---wouldn't do
instead.  For example, it might be convenient for implementation if
{\tt Procs} was HPspmd but it's base class {\tt Group} wasn't.]}
Ordinary Java code (not processed by the HPJava translator) {\em may}
access members of HPspmd classes, but this requires detailed knowledge of
the transformations the HPJava translator applies to function
signatures%
\footnote{We will use the term {\em function} to cover both
methods and constructors.}
and distributed array variables, and in general this facility should be
used very cautiously.  The HPJava translator itself will prevent code
in non-HPspmd classes from creating instances of, or using members
from, HPspmd classes.  Of course there is no restriction in the other
direction---an HPspmd class can freely use any non-HPspmd Java class,
limited only by the normal accessibility rules.

The HPJava translator will also not allow an HPspmd class to override
methods from the base class {\tt Object}.  For example, an HPspmd
class cannot define a {\tt finalize()} method.  This is just as well,
because the {\tt finalize()} method is invoked asynchronously by the
garbage collector.  In this context there is no well-defined active
process group, and any attempt to, say, invoke a collective operation
from a {\tt finalize()} method would almost certainly be a disaster.

It should be very clear that the distinction between HPspmd and non-HPspmd
classes is orthogonal to the discussion in section \ref{sec:arrayTypes}
about the types of distributed arrays themselves.  Here we are
essentially concerned with what kind of code is allowed to {\em use}
distributed arrays.  HPspmd classes are unequivocally Java classes,
whereas distributed array types are not classes of any kind.

\clearpage 

%\section{Syntax and parsing}
%
%{\em [\ldots]}
%
%\subsection{A Note on Compilation of Distributed Array Constructors}
%
%The group parameter in an ``{\tt on} clause'' can be any group-valued expression.
%Consider this unusual but legal program fragment%
%\footnote{We pointed out in section \ref{sec:otherDistributions} that
%{\tt Dimension} is actually a subclass of {\tt Range}.  So the
%expression {\tt d[0]} is well-defined, and its value is a location.}%
%:
%\begin{displaymath}
%\begin{code}
%\begin{verbatim}
%  Procs2 p = new Procs2(2, 2);
%
%  Range x = new BlockRange(5, p.dim(0));
%  Range d = p.dim(1);
%
%  float [[-]] a = new float [[x]] on p / d [0];
%\end{verbatim}
%\end{code}
%\end{displaymath}
%The mapping of {\tt a} is illustrated in Figure \ref{fig:restrictArray}.
%
%\begin{figure*}[tbp]
%\centerline{\psfig{figure=restrictArray.eps,width=2in}}
%\caption{One-dimensional array with restricted
%distribution.\label{fig:restrictArray}}
%\end{figure*}
%
%There is an ambiguity in how the allocation expression
%\begin{displaymath}
%\begin{code}
%\begin{verbatim}
%    new float [[x]] on p / d [0]
%\end{verbatim}
%\end{code}
%\end{displaymath}
%is parsed.  The most obvious interpretation is probably:
%\begin{displaymath}
%\begin{code}
%\begin{verbatim}
%    new float [[x]] on (p / d [0])
%\end{verbatim}
%\end{code}
%\end{displaymath}
%with the {\tt /} operator representing a group restriction.
%However it might also be parsed as:
%\begin{displaymath}
%\begin{code}
%\begin{verbatim}
%    (new float [[x]] on p) / d [0]
%\end{verbatim}
%\end{code}
%\end{displaymath}
%with the {\tt /} operator representing an {\em array} restriction.
%
%Actually this syntactic ambiguity doesn't affect the meaning of the
%program.  In the second case the subexpression
%\begin{displaymath}
%\begin{code}
%\begin{verbatim}
%    new float [[x]] on p
%\end{verbatim}
%\end{code}
%\end{displaymath}
%represents a new array replicated over the dimension {\tt d}.  Restricting
%this array to {\tt d[0]} yields a result essentially
%indistinguishable from the one produced by allocating a new array within
%the subgroup $\mbox{\tt p} / \mbox{\tt d[0]}$.  
%
%As a practical matter the freedom to choose the second interpretation
%eases the job of the parser, because the concrete syntax for the
%{\tt on} clause can be simplified to:
%\begin{displaymath}
%\begin{code}
%\begin{verbatim}
%    <OnClause> ::= on <PrimaryExpression>
%\end{verbatim}
%\end{code}
%\end{displaymath}
%instead of the ambiguous:
%\begin{displaymath}
%\begin{code}
%\begin{verbatim}
%    <OnClause> ::= on <Expression>
%\end{verbatim}
%\end{code}
%\end{displaymath}
%A later phase of the compiler may recognize the special case of
%a restricted array allocation, and convert it to the more efficient
%form using group-restriction.
%
%{\em [Maybe this needs to be worded more carefully, because {\em Primary}
%has a strange definition in the Java Language Specification.]}
%
%\clearpage 


\clearpage

\section{Static Semantic Checking}

\subsection{Subscripting}
\label{sec:subscriptChecking}

In section \ref{sec:elementAccessRules} we gave a handful of special rules
for distributed array element access.  One example is that in
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$    overall(i = $$x$\verb$ for $$l$\verb$ : $$u$\verb$ : $$s$\verb$) {$ \\
\verb$      ... $$a \mbox{\tt [}e_0\mbox{\tt ,} \ldots
                    \mbox{\tt ,} e_{r - 1}
                    \mbox{\tt ,} i
                    \mbox{\tt ,} e_{r + 1} \mbox{\tt ,} \ldots
                    \mbox{\tt ]}$ \verb$ ...$ \\
\verb$    }$
\end{tabbing}
\end{bigCode}
\end{displaymath}
the expression $a$ must be invariant in the {\tt overall} construct.

In general, proving that an arbitrary expression is invariant throughout
a particular block of code is an intractable problem for a compiler.
The block of code (our {\tt overall} construct) may, for example, contain
method invocations.  If the expression we are analysing involves fields
or arrays, it is not generally possible to be prove that the method
invocation does not change the value of the expression as a side-effect.
On the other hand, if one could {\em not} prove that expressions like $a$
above were invariant, the loss of efficiency in the translated code
might be quite spectacular.

For these reasons the translator imposes the following, fairly sweeping,
restrictions:
\begin{itemize}
\item
{\em Every} distributed-array-valued variable in an HPJava program implicitly
has the {\tt final} attribute.

Note that Java (and HPJava) support an idea of {\em blank final}
variables.  A blank final variable is a variable that is declared with
the {\tt final} attribute, meaning it is essentially constant, but its
value is not initialized in its declaration.
Instead it is initialized in a later assignment.
Thus a blank final variable is a kind of {\em single-assignment
variable}.  
A blank final {\em instance variable}, for example,
can (and in fact must) be initialized in a constructor of the class to
which it belongs.  
Restricting distributed array variables to be final variables
is thus not as limiting as it may sound, because these array
variables may be blank finals.  On the other hand this restriction allows
accesses to the special arrays of HPJava to be analysed with at least
some of the simplicity of earlier, more static, languages like Fortran,
and this is a very important advantage.  Moreover we suspect that this
limitation will rarely be noticed by the programmer---it merely enforces
practices that are likely to be commonplace.

\item
To take full advantage of this limitation, we impose an additional
ad hoc rule that the array reference expression in a distributed array
element reference or an array section expression {\em must be a simple
identifier}.

This identifier can refer to a local variable or an instance variable of
the current object.  Again the
actual inconvenience to the programmer due to this restriction is
fairly minor.  It seems only to enforce common practices.  If however,
a method really {\em does} need to process a distributed-array-valued
field in another object (say) the reference just needs to be copied to
a distributed-array-valued {\em local variable} prior to processing the
elements (or computing the section).
\end{itemize}

We can now very cleanly enforce the original requirements on subscripting
in {\tt overall} and {\tt at} constructs, by applying a simple static
check that if a distributed index {\em i} appears in a subscript of a
distributed array {\em a}, the variable {\em a} should have been declared
{\em outside} the scope of {\em i}, and (if it is a blank final) $a$
should not be assigned {\em inside} the scope of {\em i}.

\subsection{Initialization of fields}
\label{sec:fieldInitializers}

Any HPspmd code needs to execute in the context of a well-defined active
process group (APG).  In the current implementation of HPJava, the APG is
defined only in the bodies of methods and constructors of HPspmd classes.
There is no APG in effect for code that executes outside the bodies of
these functions.  So in particular executable HPspmd code (meaning
expressions that invoke methods of HPspmd classes, or use the distributed
arrays of HPJava) is not allowed in initializers for fields or
class initialization blocks, including {\tt static} initialization blocks.

This means in particular that:
\begin{quote}
Fields whose type is an HPspmd class type or an HPspmd interface
type (including the built-in
{\tt Group} hierarchy of class types) cannot have initializers in
their declarations.  
\end{quote}
Similarly:
\begin{quote}
Fields that have distributed array type cannot have initializers in
their declarations.  
\end{quote}
Since distributed-array-valued fields, like all distributed-array-valued
variables, are implicitly {\tt final},
it follows that they are also implicitly {\em blank final}.  In the
case of instance variables, this means they must be initialized in the
constructors of the class.  In general Java requires that blank final
{\em static} fields be initialized in static initialization blocks.
Since these blocks cannot contain executable HPspmd code, it follows
that a distributed-array-valued {\em static} field could never be initialized.
Consequently:
\begin{quote}
HPJava forbids distributed-array-valued fields from having
the {\tt static} attribute.
\end{quote}


\clearpage

\section{Pre-translation}

The current HPJava translator has (at least) two translation phases.  The first
phase, {\em pre-translation}, reduces the input HPJava program to some
other, equivalent HPJava program, coded in a restricted subset of the
full HPJava language.  The translation phase proper follows
pre-translation, and converts the intermediate HPJava program to a
standard Java program.

Roles of the pre-translator include:
\begin{itemize}
\item
As explained above, the input program is reduced to a {\em restricted
form} that is suitable for processing in the translation phase proper.
This restricted form, which is described in the next section, disallows
certain complex expressions involving distributed arrays.  The
pre-translator reduces these complex expressions to simpler
expressions, by introducing temporaries if necessary.
\item
To do the required transformations in an effective way, the pre-translator
will perform some scalar dependence analysis on the source program.
The pre-translator may also annotate (the translator's internal
representation of) the program with information obtained in this analysis,
for use by later phases.
\item
The pre-translator may insert code for {\em run-time checking} of various
HPspmd rules described in chapter \ref{sec:rules}.
\end{itemize}


\subsection{Restricted Form}
\label{sec:restrictedForm}

So far as the HPJava translator is concerned, a program is in {\em
restricted form} if it respects certain limits on the complexity of
expressions involving distributed arrays.  First we give two definitions:
\begin{itemize}
\item
{\em Constructive distributed array expressions} are:
\begin{description}
\item{\bf a)}
distributed array creation expression,
\item{\bf b)}
distributed-array-valued method invocation expression,
\item{\bf c)}
array section expression, and
\item{\bf d)}
array restriction expression.
\end{description}
\item
An expression is a {\em simple expression} if it is either a
constant expression or a simple variable name (an identifier).
If it is a variable name, the named variable must not be volatile.
\end{itemize}

Now the restrictions on expressions are:
\begin{enumerate}
\item
A constructive distributed array expression may {\em only} appear as the
right-hand-side of a top-level assignment.  Here, a {\em top-level
assignment} is defined to be an assignment expression appearing as a
{\em statement expression} in a program block.  Note that this specifically
excludes assignments appearing in for-loop headers.  Also note that a
variable declarator with an initializer is {\em not} an assignment
expression, so the restricted form does not allow constructive
distributed array expressions to appear as initializers in declarations.
\item
A group restriction operation may only appear as the
right-hand-side of a top-level assignment.  The same comments apply.
\item
All of the following {\em must} be simple expressions:
\begin{description}
\item{\bf a)}
the target object in accesses to distributed-array-valued fields,
\item{\bf b)}
the boolean expression in a distributed-array-valued conditional expressions,
and
\item{\bf c)}
the array operand in array-restriction expression,
\item{\bf d)}
the array operand of distributed array inquiry operations ({\tt rng()},
{\tt grp()}, etc)%
\footnote{There is one situation in which a general
non-constructive array expression would cause a problem here, and that
is expressions like {\tt (a = e).grp()}, where the translation scheme
of Figures \ref{fig:transArrayInquiries1} and \ref{fig:transArrayInquiries2}
would cause assignments to some other components
of $a$ to be improperly discarded.  
If preferred, this situation could be addressed by a more specific rule.}%
,
\item{\bf e)}
the range and group parameters of distributed array creation expressions,
\item{\bf f)}
the range expression in the header of {\tt overall} and {\tt at} constructs.
\end{description}
\item
The group expression in the header of an {\tt on} construct must be a simple
expression {\em that is invariant in the body of the construct}.
\item
The index expression in the header of an {\tt at} construct must be
a simple expression that is invariant in the body of the construct.
\end{enumerate}
In principle reducing a program to restricted form is a straightforward
matter of precomputing subexpressions that break the rules above,
and saving their values in temporary variables introduced by the
pre-translator.  Of course the subexpressions of the original
expression are replaced by accesses to the temporary%
\footnote{As a special case, an invocation of a method that returns a
distributed array, but which appears alone as a statement expression
(with the result value ignored), may be pre-translated to an assignment
to a temporary variable whose value is subquently ignored.}%
.

One problematic area is arguments and prefixes in
explicit constructor invocations.  An explicit constructor invocation%
\footnote{Which should not be confused with the implicit constructor
invocation in an object creation expression.} must be the first statement
in the body of a constructor, so we cannot place an assignment to
a temporary before it.  The easiest way to handle this situation is to add
an ad hoc rule limiting the complexity of expressions that may appear
in this context in the HPJava source program---for example we may insist
that any distributed-array-valued expressions appearing as a subexpression
in an explicit constructor invocation statement must be a simple
expression.

Figures \ref{fig:egPreTransSource} and \ref{fig:egPreTransOut} illustrate
the process of expression simplification.  The source program is a
slightly esoteric example involving array restriction, taken from section
\ref{sec:arrayRestriction}.  In this example pre-translation shifts three
kinds of constructive array expression---distributed array creation,
array section and array restriction---into top-level assignments.
In the case of the array creation expressions it simply has to split
the array declarations with initializers to blank declarations plus
an assignment.  In the case of the sections and restrictions, it has to
introduce temporaries.

\begin{figure}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs3 p = new Procs3(P, P, P) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;
    Range y = new BlockRange(N, p.dim(1)) ;
    Range z = new BlockRange(N, p.dim(2)) ;

    float [[-,-]] c = new float [[x, y]] ;

    float [[-,-]] a = new float [[x, z]] ;
    float [[-,-]] b = new float [[z, y]] ;

    ... initialize `a', `b'

    overall(i = x for :)
      overall(j = y for :)
        c [i, j] = Adlib.dotProduct(a [[i, :]] / j, b [[:, j]] / i) ;
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Example source program prior to pre-translation.%
\label{fig:egPreTransSource}}
\end{figure}

\begin{figure}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs3 p = new Procs3(P, P, P) ;

  on(p) {
      Range x = new BlockRange(N, p.dim(0)) ;
      Range y = new BlockRange(N, p.dim(1)) ;
      Range z = new BlockRange(N, p.dim(2)) ;

      float [[-,-]] c ;
      c = new float [[x, y]] ;

      float [[-,-]] a ;
      a = new float [[x, z]] ;
      float [[-,-]] b ;
      b = new float [[z, y]] ;

      ... initialize `a', `b'

      overall(i = x for :)
          overall(j = y for :) {
              float [[-]] __$t1 ;
              __$t1 = a [[i, :]] ;

              float [[-]] __$t2 ;
              __$t2 = __$t1 / j ;

              float [[-]] __$t3 ;
              __$t3 = b [[:, j]] ;

              float [[-]] __$t4 ;
              __$t4 = __$t3 / i ;

              c [i, j] = Adlib.dotProduct(__$t2, __$t4) ;
          }
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Example source program after pre-translation.%
\label{fig:egPreTransOut}}
\end{figure}

Besides simplifying variables, the pre-translation phase also
breaks up distributed array declarations that declare several
variables, outputting a series of declarations of {\em individual}
distributed array variables.  This is slightly more convenient
for the translation phase.

\subsection{Expression simplification}
\label{sec:expressionSimplification}

In principle the transformations needed to simplify expressions and
reduce a program to restricted form are relatively straightforward.
For completeness we will outline in some detail the algorithm applied
by the current HPJava translator.  We should remark that much of the
discussion in this section is not very specific to HPJava; similar
strategies may be used in other Java preprocessors that need to
conservatively simplify expressions, as a pre-requisite to translating
some subset of those expressions.

One principle of the HPspmd model is that parts of a program that
are written entirely in the base language---not using special HPspmd
features---should, as far as practical, be compiled in exactly the same
way as if they appeared in a local sequential program.
We will take this requirement as a guiding principle in defining our
algorithm for simplifying expressions.  Sub-expressions that are pure
Java expressions---not involving distributed arrays---should, as
far as possible, be copied unchanged to the translated program.
Ideally the generated Java program should not break up
these subexpressions by introducting temporaries.
On the other hand, a ``pure Java'' sub-expression may as a whole be
precomputed and stored in a temporary, if it was originally embedded in
an expression that included special HPJava terms elsewhere.

In our scheme certain subexpressions are singled out according to
the simplification rules.  These are expressions that {\em must}
be precomputed, to meet the requirements of a subsequent translation
phase.  In general, if the these subexpressions are precomputed,
dependence relations may force some other, ``ordinary'' subexpressions to
be precomputed as well.

\subsubsection{Special subexpressions}
\label{sec:specialSubexpressions}

We will identify three classes of expressions that need special
treatment: {\em composite expressions}, {\em multiply referenced values},
and {\em multiply referenced variables}.
\begin{itemize}
\item
A composite expression is an expression that requires some preliminary
statements to be executed before they the expression can be evaluated.
Typically these are special expressions of the extended language---for
example they may need some procedure call to proceed their evaluation.
In general, the
translation scheme will treat composite expressions differently according to
whether they appear as the right-hand side of a top-level assignment,
or anywhere else.  In the former case it not usually necessary for the
{\em pre-translation phase} to introduce temporaries.
\item
A multiply referenced value is any expression---possibly an ordinary
Java expression---that appears in a context that means the translation
phase proper to will need to use the expression's value more than once.
Often these are subexpressions of composite expressions.
\item
A multiply referenced {\em variable} is an expression that has the
syntax of a program variable, and appears in a context that means the
translation scheme will use the expression more than once,
{\em with at least one of these uses being a definition} (for example,
the expression may appear as the left-hand side of an assignment).
\end{itemize}
The singled-out expressions in the current HPJava translation scheme
are listed in Figure \ref{fig:specialExpressions}.  Note that this
table largely encapsulates the HPJava-specific features of the
pre-translation scheme described in this section.

\begin{figure}[tbp]
\begin{center}
\begin{minipage}[t]{4in}
{\em Composite expressions:}
\begin{enumerate}
\item
Constructive array expressions.
\item
Group restriction operations.
\end{enumerate}
{\em Multiply referenced values:}
\begin{enumerate}
\item
The target object in accesses to distributed-array-valued fields.
\item
The boolean expression in a distributed-array-valued conditional expressions.
\item
The distributed array operand in array-restriction expression.
\item
The array operand of distributed array inquiry operations ({\tt rng()},
{\tt grp()}, etc).
\item
The range and group parameters of distributed array creation expressions.
\item
The range expression in the header of {\tt overall} and {\tt at} constructs.
%\item
%The group expression in the header of an {\em on} construct.
%\item
%The index expression in the header of an {\em at} construct.
\end{enumerate}
\end{minipage}
\end{center}
\caption{Special expressions in the current HPJava translation scheme.}
\label{fig:specialExpressions}

\end{figure}

\subsubsection{Schematic algorithm}

A generic version of the algorithm for simplification is given in Figure
\ref{fig:exprSimplification}.  The algorithm is written as a recursive
function, {\it simplify}.  The main input to this function is an
expression term $e$, which is modified by the function.
Along with the modified version of $e$, the {\it simplify} function
outputs a sequence of statements,
{\it INITS}, which declares and initializes any temporaries that
were introduced in the simplification of $e$.  It also outputs
``access sets'' for $e$ and {\tt INITS}.

\begin{figure}[tbp]
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
{\it simplify}\verb$($$e$\verb$, ${\it INITS}\verb$, ${\it ACCESS}\verb$, ${\it INITS\_ACCESS}\verb$) {$ \\
\verb$    $\=\verb$INOUT: $$e$ \\
           \>\verb$OUT: ${\it INITS}\verb$, ${\it ACCESS}\verb$, ${\it INITS\_ACCESS} \\
\\
      \>{\it INITS}\verb$ = $``'' \\
\\
      \>{\it ACCESS}\verb$ = $$\{\}$ \\
      \>{\it INITS\_ACCESS}\verb$ = $$\{\}$ \\
\\
      \>\verb$foreach ($subexpression $e_i$ of $e$, enumerated
                      {\em right to left}\verb$) {$ \\
\\
      \>\verb$    $\={\it simplify}\verb$($$e_i$\verb$, $$\mbox{\it INITS}_i$\verb$, $$\mbox{\it ACCESS}_i$\verb$, $${\it INITS\_ACCESS}_i$\verb$)$ \\
\\
      \>    \>\verb$if($\=$e_i$ is a multiply referenced value that is not simple, {\em or} \\
      \>    \>          \>there is a dependency between $\mbox{\em ACCESS}_i$ and {\em INITS\_ACCESS}\verb$) {$ \\
\\
      \>    \>\verb$    $\=\verb$// $$t$ is a new temporary name; $T$ is type of $e_i$ \\
\\
      \>    \>    \>{\it INITS}\verb$ = $$\mbox{\it INITS}_i$\verb$ + $``\verb$ $$T$\verb$ $$t$\verb$ ; $$t$\verb$ = $$e_i$\verb$ ; $''\verb$ + ${\it INITS} \\
      \>    \>    \>$e_i$\verb$ = $``$t$'' \\
\\
      \>    \>    \>{\it INITS\_ACCESS}\verb$ = $$\mbox{\it INITS\_ACCESS} \,\, \cup \mbox{\it INITS\_ACCESS}_i \,\, \cup \mbox{\it ACCESS}_i$ \\
      \>    \>\verb$} else {$ \\
      \>    \>    \>{\it INITS}\verb$ = $$\mbox{\it INITS}_i$\verb$  + ${\it INITS} \\
\\
      \>    \>    \>{\it ACCESS}\verb$ = $$\mbox{\it ACCESS} \,\, \cup \mbox{\it ACCESS}_i$ \\
      \>    \>    \>{\it INITS\_ACCESS}\verb$ = $$\mbox{\it INITS\_ACCESS} \,\, \cup \mbox{\it INITS\_ACCESS}_i$ \\
      \>    \>\verb$}$ \\
      \>\verb$}$ \\
\\
      \>{\it ACCESS}\verb$ = $$\mbox{\it ACCESS} \,\, \cup \, \, \{\mbox{direct accesses associated with the node $e$}$\} \\
\\
      \>\verb$if($\=$e$ is a composite expression {\em and} \\
      \>          \>the parent node of $e$ is not a top-level assignment\verb$) {$ \\
\\
      \>\verb$    $\=\verb$// $$t$ is a new temporary name; $T$ is type of $e$ \\
\\
      \>      \>{\it INITS}\verb$ = ${\it INITS}\verb$ + $``\verb$ $$T$\verb$ $$t$\verb$ ; $$t$\verb$ = $$e$\verb$ ;$'' \\
      \>      \>$e$\verb$ = $``$t$'' \\
\\
      \>      \>{\it INITS\_ACCESS}\verb$ = $$\mbox{\it INITS\_ACCESS} \,\, \cup \mbox{\it ACCESS}$ \\
      \>      \>{\it ACCESS}\verb$ = {}$ \\
      \>\verb$}$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
\caption{Schematic algorithm for expression simplification.}
\label{fig:exprSimplification}

\end{figure}

The algorithm traverses sub-expressions in {\em right to left} order.
If a sub-expression must be precomputed, the associated code will have
to be moved in front of sibling sub-expressions currently to its left.
If there are no dependencies between the moved sub-expression and its
left siblings, this is fine.  But if there is a dependency, the sibling
expression must also be precomputed, to preserve the original order
of evaluation.

Dependencies are detected using {\em access sets}.  These record uses
and definitions of individual variables, or categories of variables, within
expressions.  The {\em simplify} function computes these access sets on
the fly and returns two of them in the values {\it ACCESS} and
{\it INIT\_ACCESS}.  The set {\it ACCESS} is the set of program variable
accesses in the translated expression $e'$.  The set {\it INIT\_ACCESS}
is the set of program variable accesses in the ``precomputation''
code---code for initialization of temporaries defined in {\em INITS}
(both sets exclude accesses to temporaries themselves).

At the time a subexpression is visited, the variable {\it INIT\_ACCESS}
holds accesses in ``precomputation'' code already generated by
subexpressions {\em to the right} of the current expression.  If a
dependency is discovered between the translated version of the {\em
current} expression and the code we already know must move ``to
the left'', the whole of the current expression must also ``move to
the left'', to be precomputed before the code already scheduled to move.

Multiply referenced values and composite expressions are handled
in different parts of the algorithm, but currently they are a treated in
exactly the same way.  It would be easy to unify the treatment
of these two cases.  There are two reasons not to do so.  First, the
version given in Figure \ref{fig:exprSimplification} is closer to a
practical organization---with this organization the information needed to
determine whether to precompute is closely associated with the current
node $e$.  Secondly, this version should adapt more easily to future
versions of the pretranslation/translation strategy, in which the
initialization code for composite nodes might be more refined than
wholesale copying to a temporary.

\subsubsection{Data structures}
\label{sec:accessSetData}

In the pseudocode of Figure \ref{fig:exprSimplification} code fragments
are written as Java-like strings of program text (with \verb$+$ as the
concatenation operator).  In a practical implementation these would
be manipulated as nodes in an abstract syntax tree.

Access sets will probably be implemented as two separate variable sets: the
set of {\em uses} and the set of {\em defs}.  A straightforward
(possibly inefficient) approach is to implement each of these two sets
using four components:
\begin{enumerate}
\item
A hash set of local variable descriptors.
\item
A hash set of field descriptors (which may be considered to
represent ``all instances'' of the field involved, if they are
instance variable descriptors).
\item
A flag specifying that ``all fields'' must be assumed to be included in
the set.  If this flag is {\tt true}, the previous component can be ignored.
\item
A flag specifying ``all array elements'' must be assumed to be
included in the set.
\end{enumerate}

We consider two access sets to be {\em dependent} if they share
a common variable (or category of variables), and at least one of the
accesses is a {\em def}.

To complete the {\em simplify} algorithm we still need to define, for each
kind of expression in the language, the relevant set of subexpressions
(enumerated in the {\tt foreach}), and also to define the ``direct accesses
associated with'' this kind of expression.

\subsubsection{Expression nodes}

To apply the {\em simplify} algorithm we need to define the relevant
sub-expressions of, and ``direct accesses'' in, the various kinds
of expression node in the language.  The non-trivial cases---the cases
that have non-empty sets of direct accesses---are:
\begin{itemize}
\item
Updates:
\begin{itemize}
\item
simple assignment
\item
auto-increment/decrement (\verb$++$, etc)
\item
compound assignment (\verb$+=$, etc)
\end{itemize}
\item
Variable expressions:
\begin{itemize}
\item
named expression
\item
field reference with expression prefix
\item
field reference with {\tt super} prefix
\item
array element reference (distributed array or Java array)
\end{itemize}
\item
``Wildcards'':
\begin{itemize}
\item
method invocations
\end{itemize}
\end{itemize}
The ``variable'' expressions defined here will be treated differently,
according to whether they appear as the variable operand of an update,
or anywhere else.  A ``named expression'' is any expression with the
syntax of one or more identifiers separated by periods.

Table \ref{fig:simplifyAccesses} gives suitable definitions for the
``direct accesses'' in these nodes.  In these tables LHS stands
for the left-hand-side operand of an assignment; RHS stands for the
right-hand-side operand.

\begin{table}

\begin{center}
\begin{tabular}{|p{1.8in}|p{1.6in}|p{1.6in}|}
\hline
{\bf Expression node} & {\bf Direct {\em uses}} & {\bf Direct {\em defs}} \\
\hline
\hline
Simple assignment with named expression on LHS &
&
Variable descriptor (local or field) associated with LHS. \\
\hline
Simple assignment with field reference as LHS &
&
Field descriptor associated with LHS. \\
\hline
Simple assignment with array element on LHS &
&
``All array elements''. \\
\hline
\hline
Compound assignment with named expression as LHS &
Variable descriptor (local or field) associated with LHS. &
Variable descriptor (local or field) associated with LHS. \\
\hline
Compound assignment with field reference as LHS &
Field descriptor associated with LHS. &
Field descriptor associated with LHS. \\
\hline
Compound assignment with array element on LHS &
``All array elements''. &
``All array elements''. \\
\hline
\hline
Autoincrement/decrement with named expression as operand &
Variable descriptor (local or field) associated with operand. &
Variable descriptor (local or field) associated with operand.  \\
\hline
Autoincrement/decrement with field reference as operand &
Field descriptor associated with operand. &
Field descriptor associated with operand. \\
\hline
Autoincrement/decrement with array element as operand &
``All array elements''. &
``All array elements''. \\
\hline
\hline
Named expression, not the variable of an update. &
Variable descriptor (local or field) associated with named expression. &
\\
\hline
Field reference, not the variable of an update. &
Field descriptor associated with field reference. &
\\
\hline
Array element reference, not the variable of an update. &
``All array elements''. &
\\
\hline
\hline
Any method invocation &
``All fields''; ``All array elements''. &
``All fields''; ``All array elements''. \\
\hline
\end{tabular}
\end{center}

\caption{Direct accesses of various expression nodes, for {\it simplify}\
algorithm.}
\label{fig:simplifyAccesses}
\end{table}

No other kinds of expression involve direct accesses, and all other
kinds of expression have reasonably obvious sub-expression lists.
For example, a binary
expression node has two subexpressions and no direct accesses; 
a triplet subscript $l$\verb$ : $$u$\verb$ : $$s$ contributes three
three subexpressions to a section expression;
an (ordinary Java) array initializer term is treated as a kind of expression
with a variable number of subexpressions (and no direct accesses); etc, etc.

By suitably redefining the sets of ``subexpressions''
for update nodes, the generic form of the {\em simplify}
algorithm can be applied to all kinds of expression (except for some awkward
compound assignments discussed later).  It seems more practical, however, to
give customized versions of the {\em simplify} code for these cases.
First Figure \ref{fig:visitChildren} defines an abstracted
{\em visitChildren} algorithm, convenient particularly for variable
expressions.
The subexpressions of a variable expression are mostly self-explanatory.
They are:
\begin{itemize}
\item
If the variable is a named expression, the subexpressions are:
\begin{itemize}
\item
the prefix of named expression, if it is non-null and is an expression (not a type).
\end{itemize}
\item
If the variable is a field reference with a general expression prefix, the
subexpressions are:
\begin{itemize}
\item
the prefix expression of the field reference.
\end{itemize}
\item
If the variable is a field reference with a {\tt super} prefix, there
are no subexpressions.
\item
If the variable is an array element reference, the subexpressions are:
\begin{itemize}
\item
the target array of the element reference reference,
\item
the integer subscripts, if any, of the element reference, and
\item
the ``shift'' expressions in any shifted-distributed-index 
subscripts of the element reference.
\end{itemize}
\end{itemize}

\begin{figure}[tbp]
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
{\it visitChildren}\verb$($$e$\verb$, ${\it INITS}\verb$, ${\it ACCESS}\verb$, ${\it INITS\_ACCESS}\verb$) {$ \\
\verb$    $\=\verb$INOUT: $$e$ \\
           \>\verb$OUT: ${\it INITS} \\
           \>\verb$INOUT: ${\it ACCESS}\verb$, ${\it INITS\_ACCESS} \\
\\
      \>{\it INITS}\verb$ = $``'' \\
\\
      \>\verb$foreach ($subexpression $e_i$ of $e$, enumerated
                      {\em right to left}\verb$) {$ \\
\\
      \>\verb$    $\={\it simplify}\verb$($$e_i$\verb$, $$\mbox{\it INITS}_i$\verb$, $$\mbox{\it ACCESS}_i$\verb$, $${\it INITS\_ACCESS}_i$\verb$)$ \\
\\
      \>    \>\verb$if($\=$e_i$ is a multiply referenced value that is {\em not} simple, {\em or} \\
      \>    \>          \>there is a dependency between $\mbox{\em ACCESS}_i$ and {\em INITS\_ACCESS}\verb$) {$ \\
\\
      \>    \>\verb$    $\=\verb$// $$t$ is a new temporary name; $T$ is type of $e_i$ \\
\\
      \>    \>    \>{\it INITS}\verb$ = $$\mbox{\it INITS}_i$\verb$ + $``\verb$ $$T$\verb$ $$t$\verb$ ; $$t$\verb$ = $$e_i$\verb$ ; $''\verb$ + ${\it INITS} \\
      \>    \>    \>$e_i$\verb$ = $``$t$'' \\
\\
      \>    \>    \>{\it INITS\_ACCESS}\verb$ = $$\mbox{\it INITS\_ACCESS} \,\, \cup \mbox{\it INITS\_ACCESS}_i \,\, \cup \mbox{\it ACCESS}_i$ \\
      \>    \>\verb$} else {$ \\
      \>    \>    \>{\it INITS}\verb$ = $$\mbox{\it INITS}_i$\verb$  + ${\it INITS} \\
\\
      \>    \>    \>{\it ACCESS}\verb$ = $$\mbox{\it ACCESS} \,\, \cup \mbox{\it ACCESS}_i$ \\
      \>    \>    \>{\it INITS\_ACCESS}\verb$ = $$\mbox{\it INITS\_ACCESS} \,\, \cup \mbox{\it INITS\_ACCESS}_i$ \\
      \>    \>\verb$}$ \\
      \>\verb$}$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
\caption{Unbundled function for visiting subexpressions.}
\label{fig:visitChildren}

\end{figure}

The {\em visitChildren} function is applied to the left-hand-side of
a simple assignment in Figure \ref{fig:assignmentSimplification}.
An important point is that one does not recursively apply the {\em
simplify} algorithm to the expression representing the variable itself;
the recursion is on the {\em subexpressions} of the variable expression.
The case of an autoincrement or autodecrement expression is also
straightforward, and it is given in \ref{fig:incrementSimplification}.
There is a complication in the case of compound assignments; they
will be dicussed in the next section.  

\begin{figure}[tbp]
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
{\it simplify}\verb$($$e$\verb$, ${\it INITS}\verb$, ${\it ACCESS}\verb$, ${\it INITS\_ACCESS}\verb$) {$ \\
\verb$    $\=\verb$INOUT: $$e$ \\
           \>\verb$OUT: ${\it INITS}\verb$, ${\it ACCESS}\verb$, ${\it INITS\_ACCESS} \\
\\
\verb$    $\=\verb$// $$e$ is a simple assignment, $e_l$\verb$ = $$e_r$. \\
\\
      \>{\it simplify}\verb$($$e_r$\verb$, $$\mbox{\it INITS}$\verb$, $$\mbox{\it ACCESS}$\verb$, $${\it INITS\_ACCESS}$\verb$)$ \\
\\
      \>{\it visitChildren}\verb$($$e_l$\verb$, $$\mbox{\it INITS}_l$\verb$, $$\mbox{\it ACCESS}$\verb$, $${\it INITS\_ACCESS}$\verb$)$ \\
      \>{\it INITS}\verb$ = $$\mbox{\it INITS}_l$\verb$  + ${\it INITS} \\
\\
      \>{\it ACCESS}\verb$ = $$\mbox{\it ACCESS} \,\, \cup \{\mbox{{\em def} of LHS variable}\}$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
\caption{Simplification of simple assignment expressions.}
\label{fig:assignmentSimplification}

\end{figure}

\begin{figure}[tbp]
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
{\it simplify}\verb$($$e$\verb$, ${\it INITS}\verb$, ${\it ACCESS}\verb$, ${\it INITS\_ACCESS}\verb$) {$ \\
\verb$    $\=\verb$INOUT: $$e$ \\
           \>\verb$OUT: ${\it INITS}\verb$, ${\it ACCESS}\verb$, ${\it INITS\_ACCESS} \\
\\
      \>\verb$// $$e$ is one of $e_v$\verb$++$, $e_v$\verb$--$, \verb$++$$e_v$, or \verb$--$$e_v$. \\
\\
      \>{\it ACCESS}\verb$ = $$\{\}$ \\
      \>{\it INITS\_ACCESS}\verb$ = $$\{\}$ \\
\\
      \>{\it visitChildren}\verb$($$e_v$\verb$, $$\mbox{\it INITS}$\verb$, $$\mbox{\it ACCESS}$\verb$, $${\it INITS\_ACCESS}$\verb$)$ \\
\\
      \>{\it ACCESS}\verb$ = $\=$\mbox{\it ACCESS} \,\, \cup$ \\
      \>   \>$\{\mbox{{\em use} of operand variable}, \mbox{{\em def} of operand variable}\}$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
\caption{Simplification of autoincrement/decrement expressions.}
\label{fig:incrementSimplification}

\end{figure}

%\begin{figure}[tbp]
%\begin{displaymath}
%\begin{bigCode}
%\begin{tabbing}
%{\it simplify}\verb$($$e$\verb$, ${\it INITS}\verb$, ${\it ACCESS}\verb$, ${\it INITS\_ACCESS}\verb$) {$ \\
%\verb$    $\=\verb$INOUT: $$e$ \\
%           \>\verb$OUT: ${\it INITS}\verb$, ${\it ACCESS}\verb$, ${\it INITS\_ACCESS} \\
%\\
%\verb$    $\=\verb$// $$e$ is a {\em use} of a variable. \\
%\\
%      \>{\it ACCESS}\verb$ = $$\{\}$ \\
%      \>{\it INITS\_ACCESS}\verb$ = $$\{\}$ \\
%\\
%      \>{\it visitChildren}\verb$($$e$\verb$, $$\mbox{\it INITS}$\verb$, $$\mbox{\it ACCESS}$\verb$, $${\it INITS\_ACCESS}$\verb$)$ \\
%\\
%      \>{\it ACCESS}\verb$ = $$\mbox{\it ACCESS} \,\, \cup \{\mbox{{\em use} of this variable}\}$ \\
%\verb$}$
%\end{tabbing}
%\end{bigCode}
%\end{displaymath}
%\caption{Simplification of simple variable use.}
%\label{fig:useSimplification}
%
%\end{figure}

%\clearpage

\subsubsection{Compound assignments}

Compound assignment needs special treatment in the {\em simplify}
algorithm.  Assuming $\oplus$ is one of the supported operators, a problem
arises in
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$v$\verb$ $$\oplus$\verb$= $$e$
\end{tabbing}
\end{bigCode}
\end{displaymath}
if $e$ includes a subexpression $e_i$ that must be precomputed, {\em and}
$e_i$ may define $v$ (i.e. subexpression $e_i$ has a {\em def} of $v$ in its
access set).  The underlying problem is that there is then an
anti-dependence from $v$ to $e_i$ {\em and} an output dependence from $e_i$
to $v$.  Hence one cannot move the code for $e_i$ without splitting the
compound assignment.

In this case we are essentially forced to translate the assignment to
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$v$\verb$ = $$v$\verb$ $$\oplus$\verb$ $$e$
\end{tabbing}
\end{bigCode}
\end{displaymath}
thus allowing the use of $v$ on the RHS to be precomputed.  Moreover if
$v$ itself is a non-trivial expression, you may then have to mark its
subexpressions as being multiply-referenced to avoid their repeated
computation.

It is fairly difficult to imagine realistic code in which this situation
would arise, but here is a contrived example:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    a [n++] += sum(blackBox()) ;
\end{verbatim}
\end{code}
\end{displaymath}
We will suppose {\tt blackBox()} is a method that returns a distributed
array result, but whose behavior is otherwise unknown.
The method {\tt sum()} might add the elements of a distributed array.
On the left-hand-side, {\tt a} is an array---a local variable,
say---and we will assume {\tt n} is also a local variable.

According to our simplification rules, the constructive distributed
array expression {\tt blackBox(a)} must be precomputed, and assigned to
a temporary.  So a naive transformation might be:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    __$t1 = blackBox() ;
    a [n++] += sum(__$t1) ;
\end{verbatim}
\end{code}
\end{displaymath}
where {\tt \_\_\$t1} is the name of a temporary.  In general this
transformation is illegal, because {\tt blackBox()} could modify
the value of {\tt a [n]} as a side-effect (even though
the array reference {\tt a} is a local variable, the referenced array 
may be accessible by other methods).
To solve this problem, we should precompute the initial value of the
left-hand-side of the assignment, something like
\begin{displaymath}
\begin{code}
\begin{verbatim}
    __$t1 = a [n++] ;
    __$t2 = blackBox() ;
    a [n++] = __$t1 + sum(__$t2) ;
\end{verbatim}
\end{code}
\end{displaymath}
Of course this still isn't correct due to the repetition
of the subscript expression.  A correct transformation is:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    __$t1 = n++ ;
    __$t2 = a [__$t1] ;
    __$t3 = blackBox() ;
    a [__$t1] = __$t2 + sum(__$t3) ;
\end{verbatim}
\end{code}
\end{displaymath}

An algorithm is given in Figure \ref{fig:compoundSimplification}.
Note that $e_l$ may be marked as a {\em multiply referenced variable}.
Such expressions were mentioned in section
\ref{sec:specialSubexpressions}.  The practical effect of marking a
variable expression in this way will be to alter the behavior of the {\em
visitChildren} function defined in Figure \ref{fig:visitChildren}, causing
it to treat all the subexpressions, $e_i$, as multiply referenced {\em values}.

\begin{figure}[tbp]
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
{\it simplify}\verb$($$e$\verb$, ${\it INITS}\verb$, ${\it ACCESS}\verb$, ${\it INITS\_ACCESS}\verb$) {$ \\
\verb$    $\=\verb$INOUT: $$e$ \\
           \>\verb$OUT: ${\it INITS}\verb$, ${\it ACCESS}\verb$, ${\it INITS\_ACCESS} \\
\\
\verb$    $\=\verb$// $$e$ is a compound assignment, $e_l$\verb$ $$\oplus$\verb$= $$e_r$. \\
\\
      \>{\it simplify}\verb$($$e_r$\verb$, $$\mbox{\it INITS}$\verb$, $$\mbox{\it ACCESS}$\verb$, $${\it INITS\_ACCESS}$\verb$)$ \\
\\
      \>\verb$if(${\em INITS\_ACCESS} contains a {\em def} of LHS variable\verb$) {$ \\
\\
      \>\verb$    $\=Mark $e_l$ as a ``multiply referenced variable'' \\
      \>      \>{\it visitChildren}\verb$($$e_l$\verb$, $$\mbox{\it INITS}_l$\verb$, $$\mbox{\it ACCESS}$\verb$, $${\it INITS\_ACCESS}$\verb$)$ \\
\\
      \>      \>\verb$// $$t$ is a new temporary name; $T$ is type of $e_l$ \\
\\
      \>      \>{\it INITS}\verb$ = $$\mbox{\it INITS}_l$\verb$  + $
                        `` $T$\verb$ $$t$\verb$ ; $$t$\verb$ = $$e_l\verb$ ; $$''\verb$ + $$\mbox{\it INITS}$ \\
      \>      \>$e$\verb$ = $``$e_l$\verb$ = $$t$\verb$ $$\oplus$\verb$ $$e_r$'' \\
\\
      \>\verb$} else {$ \\
      \>\verb$    $\={\it visitChildren}\verb$($$e_l$\verb$, $$\mbox{\it INITS}_l$\verb$, $$\mbox{\it ACCESS}$\verb$, $${\it INITS\_ACCESS}$\verb$)$ \\
\\
      \>\verb$    $\={\it INITS}\verb$ = $$\mbox{\it INITS}_l$\verb$  + $
                                          $\mbox{\it INITS}$ \\
      \>\verb$}$ \\
\\
      \>{\it ACCESS}\verb$ = $\=$\mbox{\it ACCESS} \,\, \cup \{\mbox{{\em use} of LHS variable}, \mbox{{\em def} of LHS variable}\}$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
\caption{Simplification of compound assignment expressions.}
\label{fig:compoundSimplification}

\end{figure}

\subsubsection{Are these transformations ``conservative''?}

Certainly we want our transformations to be conservative
in the sense that they strictly preserve the meaning of the input program%
\footnote{In our source-to-source tranlation scheme we would also like them
to be conservative in the orthogonal sense that they preserve
as much of the syntactic structure of the original program as practical.}%
.  The rules for transformation of source code we give,
based on the {\em simplify} algorithm described
above, would probably be considered conservative in ``traditional''
languages.  They would not be considered conservative in the context
of Java-to-Java transformation%
\footnote{At least not if, for some reason, some expressions in the input program were designated as multiply-referenced or composite.
A pure Java program input to the {\em actual} HPJava translator
would be effectively unchanged on output, so it is trivially conservative.}
because they do not preserve the exact model of exceptions---they allow
reorderings of code that could be detected by inspecting the values of
program variables after an exception occured.

In our framework this shortcoming can be remedied by modelling the
throwing of an exception as a {\em use} of ``all variables''.  Any
expression that could throw an exception (e.g. an arithmetic division,
a subscripting operation, a dereferencing operation, a method invocation)
would be considered to directly read the value of all variables.
The intuitive justification for this approach is that throwing an
exception can cause control to pass to a {\tt catch} or {\tt finally}
block that might read essentially any variable.  The representation
of variable sets described in section \ref{sec:accessSetData} would be
augmented by an ``all variables'' flag.

Actually we are converting HPJava to Java, not Java to Java.
HPJava is not Java, and it is not clear a priori that it must
be constrained to treat exceptions in the strict Java way.  We will let
experience decide\ldots


\subsection{Applying the {\em simplify} algorithm}

In a Java or HPJava program, expressions can appear in the context of
various kinds of statement.  This section enumerates (in much detail)
the transformations of the input program that follow from simplification
of expressions.

\subsubsection{Statement expressions}

To start with the easiest case, consider the statement
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$e$\verb$ ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $e$ is an expression.
Suppose $\mbox{\it INITS}$ and $e'$ are the results of applying
the {\em simplify} algorithm to $e$.  If $e' = e$, i.e. no
simplification is required, the input statement is unchanged.
Otherwise, it transformed to
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb${$ \\
\verb$    $\=$\mbox{\it INITS}$ \\
\>$e'$\verb$ ;$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}

The case of a statement expression appearing in the
initialization or continuation section of a {\tt for}-loop
header needs some special
treatment---see section \ref{sec:preTranslateFor}.

\subsubsection{Local variable declarations}
\label{sec:preTranslateLocalDec}

Consider a local variable declaration of the form
\begin{displaymath}
m \verb$ $ T \verb$ $ V_0 \verb$ $ I_0 \verb$, $ \ldots \verb$, $
           V_{n-1} \verb$ $ I_{n-1} \verb$ ;$
\end{displaymath}
where $m$ is an optional list of modifiers,
$T$ is a type, each term $V_i$ is a simple identifier, $v_i$,
followed by zero or more bracket pairs, \verb$[]$, 
and each optional initializer clause $I_i$ is either empty, 
or a term ``\verb$= $$e_i$'', where $e_i$ is an expression or an ordinary Java
array initializer.  We must have $n \ge 1$.  

If $T$ is {\em not} a distributed array type, we apply the {\em simplify}
algorithm to each initialization expression, $e_i$,
yielding results $\mbox{\it INITS}_i$ and $e'_i$.
If the {\em simplify} algorithm does not change any
initialization expression in the declaration, the whole declaration
is unchanged in the transformed code.  In any other case,
suppose $e_r$ is the first initialization expression that is changed
by the {\em simplify} algorithm.  We will take the transformed
declaration to be:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$m$\verb$ $$T$\verb$ $$V_0$\verb$ $$I_0$\verb$, $$\ldots$\verb$, $$V_{r-1}$\verb$ $$I_{r-1}$\verb$ ;$ \\
$D_r$ \\
\verb$...$ \\
$D_{n-1}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
If $I_i$ is empty, then $D_i$ is just:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$m$\verb$ $$T$\verb$ $$V_i$\verb$ ;$ \\
\end{tabbing}
\end{bigCode}
\end{displaymath}
If $I_i$ is not empty, then $D_i$ is:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it INITS}_i$ \\
$m$\verb$ $$T$\verb$ $$V_i$\verb$ = $$e'_i$\verb$ ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}
{\em unless} $e_i$ is a composite expression.  If $e_i$ is composite (as an
optimization, avoiding introduction of a temporary) $D_i$ is:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$m$\verb$ $$T$\verb$ $$V_i$\verb$ ;$ \\
$\mbox{\it INITS}_i$ \\
$s'_i$ ;
\end{tabbing}
\end{bigCode}
\end{displaymath}
where now
$\mbox{\it INITS}_i$ and $s'_i$ are the results of applying the {\em simplify}
algorithm to the top-level assignment expression:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$v_i$\verb$ = $$e_i$
\end{tabbing}
\end{bigCode}
\end{displaymath}

If $T$ is a distributed array type, we unconditionally break the
declaration into a series of transformed, single-variable declarations:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$D_0$ \\
\verb$...$ \\
$D_{n-1}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
with the $D_i$ are defined as above.

The case of a local variable declaration appearing in the
initialization section of a {\tt for}-loop header needs some special
treatment---see section \ref{sec:preTranslateFor}.

\subsubsection{Field declarations}

Expressions can appear as initializers in field declarations.  However,
due to restrictions imposed in section \ref{sec:fieldInitializers},
these expressions are never modified by the simplification algorithm.
So in principle field declarations need not be changed in pre-translation.
As a minor convenience to the subsequent translation phase, however,
if $T$ is a {\em distributed array} type, a field declaration:
\begin{displaymath}
m \verb$ $ T \verb$ $ V_0 \verb$, $ \ldots \verb$, $ V_{n-1} \verb$ ;$
\end{displaymath}
will be transformed to:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$m$\verb$ $$T$\verb$ $$V_0$\verb$ ;$ \\
\verb$...$ \\
$m$\verb$ $$T$\verb$ $$V_{n-1}$\verb$ ;$ \\
\end{tabbing}
\end{bigCode}
\end{displaymath}
(Recall that, according to section \ref{sec:fieldInitializers},
distributed array field declarations never have initializers.)

\subsubsection{{\tt if}, {\tt switch}, and {\tt synchronized} statements}
\label{sec:preTranslateIf}

Consider the statement
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$if($$e$\verb$) $$S$ 
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $e$ is an expression and $S$ is a statement.
As usual, $\mbox{\it INITS}$ and $e'$ are the results of applying
the {\em simplify} algorithm to $e$.  If $e' = e$, i.e. no
simplification is required, the transformed version is just
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$if($$e$\verb$) $$S'$ 
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $S'$ is the pre-translated version of $S$.
Otherwise the transformed version is
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb${$ \\
\verb$    $\=$\mbox{\it INITS}$ \\
\>\verb$if($$e'$\verb$) $$S'$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
The {\tt switch} and {\tt synchronized} statements follow exactly
the same pattern.

\subsubsection{{\tt while} statements}

We consider the statement
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$while($$e$\verb$) $$S$ 
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $e$ is an expression and $S$ is a statement. 
Again let $\mbox{\it INITS}$ and $e'$ be the results of applying
the {\em simplify} algorithm to $e$.  If $e' = e$,
the transformed version is just
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$while($$e$\verb$) $$S'$ 
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $S'$ is the pre-translated version of $S$.
Otherwise the transformed version is
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$while(true) {$ \\
\verb$    $\=\verb${$ \\
\>\verb$    $\=$\mbox{\it INITS}$ \\
\>\>\verb$if(!($$e'$\verb$)) break ;$ \\
\>\verb$}$ \\
\>$S'$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}

\subsubsection{{\tt do} statements}

We consider the statement
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$do $$S$\verb$ while($$e$\verb$) ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $e$ is an expression and $S$ is a statement. 
Again let $\mbox{\it INITS}$ and $e'$ be the results of applying
the {\em simplify} algorithm to $e$.  If $e' = e$,
the transformed version is just
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$do $$S'$\verb$ while($$e$\verb$) ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $S'$ is the pre-translated version of $S$.
Otherwise the transformed version is
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$do {$ \\
\verb$    $\=$S'$ \\
\>\=$\mbox{\it INITS}$ \\
\verb$} while($$e'$\verb$) ;$ \\
\end{tabbing}
\end{bigCode}
\end{displaymath}

\subsubsection{{\tt for} statements}
\label{sec:preTranslateFor}

The {\tt for} statement is quite complicated.  Consider
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$for($$a$\verb$; $$b$\verb$; $$c$\verb$) $$S$ 
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $a$ is either a single declaration or a (possibly empty)
comma-separated list of statement expressions, $b$ is an expression,
$c$ is a (possibly empty) comma-separated list of statement expressions,
and $S$ is a statement.

Suppose $a$ is the local variable declaration:
\begin{displaymath}
m \verb$ $ T \verb$ $ V_0 \verb$ = $ e_0 \verb$, $ \ldots \verb$, $
           V_{n-1} \verb$ = $ e_{n-1}
\end{displaymath}
where definitions are as in section \ref{sec:preTranslateLocalDec}.
Apply the transformation described in section \ref{sec:preTranslateLocalDec}
to the statement ``$a$\verb$ ;$''.  If this transformation does
not produce any changes, the transformed version of the {\tt for}
construct is:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$for($$a$\verb$; $$\mathcal{B}$\verb$; $$\mathcal{C}$\verb$) $$\mathcal{T}$ 
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $\mathcal{B}$, $\mathcal{C}$ and $\mathcal{T}$ will be defined below.
Otherwise, if the transformed version of ``$a$\verb$ ;$'' is $\mathcal{A}$,
we may take the transformed version of the {\tt for} construct to be:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb${$ \\
\verb$    $\=$\mathcal{A}$ \\
\>\verb$for( ; $$\mathcal{B}$\verb$; $$\mathcal{C}$\verb$) $$\mathcal{T}$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
(If $n = 1$, and $\mathcal{A}$ has the simple form:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it INITS}_0$ \\
$m$\verb$ $$T$\verb$ $$V_0$\verb$ = $$e'_0$\verb$ ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}
one may prefer the equivalent transformation:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb${$ \\
\verb$    $\=$\mbox{\it INITS}_0$ \\
\>\verb$for($$m$\verb$ $$T$\verb$ $$V_0$\verb$ = $$e'_0$\verb$; $$\mathcal{B}$\verb$; $$\mathcal{C}$\verb$) $$\mathcal{T}$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
This is mainly a question of aesthetics.)

Suppose, alternatively, that $a$ is the possibly-empty list of expressions:
\begin{displaymath}
a_0 \verb$, $ \ldots \verb$, $ a_{n-1}
\end{displaymath}
where $n \ge 0$.
Again, apply the {\em simplify} algorithm to each expression in this list,
and assume now that $\mbox{\it INITS}_i$ and $a'_i$ are the results of applying
the algorithm to $a_i$.
If $n = 0$ or the {\em simplify} algorithm doesn't change any
of the expressions, the transformed version of the {\tt for} construct is again:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$for($$a$\verb$; $$\mathcal{B}$\verb$; $$\mathcal{C}$\verb$) $$\mathcal{T}$ 
\end{tabbing}
\end{bigCode}
\end{displaymath}
Otherwise, we may take the transformed version of the {\tt for} construct
to be:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb${$ \\
\verb$    $\=$\mbox{\it INITS}_0$ \\
\>$a'_0$\verb$ ;$ \\
\>\verb$...$ \\
\>$\mbox{\it INITS}_{n-1}$ \\
\>$a'_{n-1}$\verb$ ;$ \\
\\
\>\verb$for( ; $$\mathcal{B}$\verb$; $$\mathcal{C}$\verb$) $$\mathcal{T}$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
Again, if $n = 1$, one might prefer the slightly simpler rendition:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb${$ \\
\verb$    $\=$\mbox{\it INITS}_0$ \\
\>\verb$for($$a'_0$\verb$; $$\mathcal{B}$\verb$; $$\mathcal{C}$\verb$) $$\mathcal{T}$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}

Now we need to define $\mathcal{B}$, $\mathcal{C}$ and $\mathcal{T}$.
We also introduce the symbols $s_b$ and $s_c$ to stand for possibly-empty
statements added at the start and end of the loop body.

Apply the {\em simplify} algorithm to $b$, and let
$\mbox{\it INITS}$ and $b'$ be the results.
If the algorithm doesn't change $b$, then 
\begin{displaymath}
\mathcal{B} = b
\end{displaymath}
and the statement $s_b$ is empty.
If the algorithm {\em does} change $b$, then
\begin{displaymath}
\mathcal{B}   = \verb$true$
\end{displaymath}
and the statement $s_b$ is:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb${$ \\
\verb$    $\=$\mbox{\it INITS}$ \\
\>\verb$if(!($$b'$\verb$)) break ;$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}

Assume $c$ is the possibly-empty list of expressions:
\begin{displaymath}
c_0 \verb$, $ \ldots \verb$, $ c_{p-1}
\end{displaymath}
Apply the {\em simplify} algorithm to all these expressions.
If the algorithm doesn't change {\em any} $c_i$, then 
\begin{displaymath}
\mathcal{C} = c
\end{displaymath}
and the statement $s_c$ is empty.
If it {\em does} change some $c_i$, then
$\mathcal{C}$ is empty and
and we will take the statement $s_c$ to be:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb${$ \\
\verb$    $\=$\mbox{\it INITS}_0$ \\
\>$c'_0$\verb$ ;$ \\
\>\verb$...$ \\
\>$\mbox{\it INITS}_{p-1}$ \\
\>$c'_{p-1}$\verb$ ;$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where now $\mbox{\it INITS}_i$ and $c'_i$ are the results of applying
the algorithm to $c_i$.

Finally, if $s_b$ and $s_c$ are both empty, the value of
$\mathcal{T}$ is just $S'$, the pre-translated version of $S$.
Otherwise the block $\mathcal{T}$ has the form
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb${$ \\
\verb$    $\=$s_b$ \\
\>$S'$ \\
\>\=$s_c$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}

\subsubsection{{\tt return} and {\tt throw} statements}

Consider the statement
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$return $$e$\verb$ ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $e$ is an expression.
As usual, $\mbox{\it INITS}$ and $e'$ are the results of applying
the {\em simplify} algorithm to $e$.  If $e' = e$, i.e. no
simplification is required, the statement is untransformed.
Otherwise the transformed version is
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb${$ \\
\verb$    $\=$\mbox{\it INITS}$ \\
\>\verb$return $$e'$\verb$ ;$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
The {\tt throw} statement follows exactly the same pattern.

\subsubsection{Explicit constructor invocation statements}

As explained in section \ref{sec:restrictedForm}, appearance of
expressions that need simplification in explicit constructor
invocation statements is problematic for a source-to-source translator.  
Hence HPJava has an ad hoc limitation saying that
these statements should not contain expressions requiring simplification.
Consequently pre-translation does not modify these statements.

\subsubsection{{\tt on} statements}

We consider the statement
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$on($$p$\verb$) $$S$ 
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $p$ is an expression and $S$ is a statement.
Translation follows the same pattern as for {\tt if}
statements, see section \ref{sec:preTranslateIf}.

A couple of special remarks apply.  In the simplification algorithm as
given in section \ref{sec:expressionSimplification}, the expression $p$ is
{\em not} required to be marked if it is a simple expression that is
{\em not modified} in the statement $S$.  This allows a rather trivial
optimization in the overall translation scheme---it may save introduction
of one temporary.  In itself this economy probably will not justify the
analysis needed to prove $p$ is invariant.  Unless detailed analysis is
planned for other reasons, one may mark $p$ unconditionally.
This forces the expression to always be saved to a temporary, declared
at the head of the construct.

In general pre-translation of the {\tt on} construct will also add some
run-time checking code.  This is discussed in section \ref{sec:runtimeChecks}.

\subsubsection{{\tt at} and {\tt overall} statements}

We consider the statement
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$at($$i$\verb$ = $$x$\verb$ [$$n$\verb$]) $$S$ 
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $x$ and $n$ are expressions and $S$ is a statement.

In a trivial way we can generalize the {\em simplify} algorithm
to apply to an ordered list of expressions.
Apply this algorithm to the ordered list $[x, n]$ and let the
results be $\mbox{\it INITS}$ and $[x',n']$.  If 
$\mbox{\it INITS}$ is empty the transformed
version of our {\tt at} statement is just
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$at($$i$\verb$ = $$x$\verb$ [$$n$\verb$]) $$S'$ 
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $S'$ is the pre-translated version of $S$.
Otherwise the transformed version is
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb${$ \\
\verb$    $\=$\mbox{\it INITS}$ \\
\>\verb$at($$i$\verb$ = $$x'$\verb$ [$$n'$\verb$]) $$S'$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}

Similar to the case of $p$ in the {\tt on} statement, the simplification
algorithm as given in section \ref{sec:expressionSimplification} does
not {\em require} the expression $n$ to be marked if it is invarient in $S$.
Again, it is fine to mark $n$ unconditionally if a proof of invariance
is not readily available.  Again, 
in general, pre-translation of the {\tt at} construct may add some
run-time checking code, as discussed in section \ref{sec:runtimeChecks}.

The {\tt overall} statement follows
the same pattern, if the triplet $t$ in
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$overall($$i$\verb$ = $$x$\verb$ for $$t$\verb$) $$S$ 
\end{tabbing}
\end{bigCode}
\end{displaymath}
is treated as a generalized kind of expression.


%Translation follows the same pattern as for {\tt if}
%statements, see section \ref{sec:preTranslateIf}.
%
%A couple of special remarks apply.  In the simplification algorithm as
%given in section \ref{sec:expressionSimplification}, the expression $p$ is
%{\em not} required to be marked if it is a the simple expression that is
%{\em not modified} in the statement $S$.  This economy is a rather trivial
%optimization of the overall translation scheme---it may save allocation
%of one temporary.  In itself this saving probably will not justify the
%analysis needed to prove $p$ is invariant.  Unless detailed analysis is
%planned for other reasons, one may choose to mark $p$ unconditionally.
%This forces the expression to always be saved to a temporary, declared
%at the head of the construct.
%
%In general pre-translation of the {\tt on} construct will also add some
%run-time checking code.  This is discussed in section \ref{sec:runtimeChecks}.


\subsection{Run-time checks}
\label{sec:runtimeChecks}

It may also be natural for the pre-translation phase to introduce
at least some of the run-time checks associated with the rules in
chapter \ref{sec:rules}.  An example is shown in Figure
\ref{fig:egPreTransChecks}.
The {\it ASSERT} macro throws an exception if its argument
is boolean {\tt false}%
\footnote{The next major release of Java is expect to include an
{\tt assert} keyword with similar functionality.}%
.  In the figure we have not yet applied the other
transformations (expression simplifications) associated with
pre-translation.

Since the pre-translation phase is supposed
to generate some {\em use-def} information for the source program 
it may be possible to eliminate many of these checks at this
early stage.  In the example given, it seems realistic to assume
that automatic analysis could detect that, for example \verb$x.dim()$
is \verb$p.dim(0)$, and that \verb$apg$ is \verb$p$ at the point the
statement
\begin{verbatim}
      ASSERT(apg.contains(x.dim()))
\end{verbatim}
appear, and thus that the check is superfluous.  In fact in this
example a moderately sophisticated analysis may well prove that all
the assertions are safe at compile-time.

\begin{figure}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
  Procs3 p = new Procs3(P, P, P) ;

  ASSERT(apg.contains(p))
  on(p) {
      Range x = new BlockRange(N, p.dim(0)) ;
      Range y = new BlockRange(N, p.dim(1)) ;
      Range z = new BlockRange(N, p.dim(2)) ;

      float [[-,-]] c = new float [[x, y]] ;

      float [[-,-]] a = new float [[x, z]] ;
      float [[-,-]] b = new float [[z, y]] ;

      ... initialize `a', `b'

      ASSERT(apg.contains(x.dim()))
      ASSERT(c.rng(0).containsLocations(x))
      ASSERT(a.rng(0).containsLocations(x))
      overall(i = x for :) {
          ASSERT(apg.contains(y.dim()))
          ASSERT(c.rng(1).containsLocations(y))
          ASSERT(b.rng(1).containsLocations(y))
          overall(j = y for :)
              c [i, j] = Adlib.dotProduct(a [[i, :]] / j, b [[:, j]] / i) ;
      }
  }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Example source program after adding run-time checks.%
\label{fig:egPreTransChecks}}
\end{figure}

%\begin{figure}[tbp]
%\textbf{SOURCE:}
%\begin{displaymath}
%\begin{bigCode}
%\begin{tabbing}
%\verb$on ($$p$\verb$) $$S$ \\
%\end{tabbing}
%\end{bigCode}
%\end{displaymath}
%
%\textbf{TRANSLATION:}
%
%\begin{displaymath}
%\begin{bigCode}
%\begin{tabbing}
%$\transpV{\egrp}$ \\
%\\
%$\mbox{\it ASSERT}($\verb$apg.contains($$p$\verb$)$$)$ \\
%\\
%\verb$on ($$p$\verb$) $$S$
%\end{tabbing}
%\end{bigCode}
%\end{displaymath}
%where:
%\begin{displaymath}
%\begin{array}{l}
%\mbox{$p$ is an expression in the source, simplified if necessary,} \\
%\mbox{$S$ is a statement in the source program,} \\
%\mbox{the {\it ASSERT} macro throws an exception if its argument
%is boolean {\tt false}.}
%\end{array}
%\end{displaymath}
%
%\caption{Insertion of run-time checking code for {\tt on} construct.}
%\label{fig:pretransOn}
%\end{figure}

\clearpage

\section{Basic translation}

After the pretranslation phase reduces a program to a suitable
restricted form, the translation phase proper begins.  This section
describes the basic, unoptimized form of this translation.

\subsection{Translation functions and schemas}

We will specify several {\em translation functions}.  The detailed
definitions will be presented through a series of {\em schema} in the
following subsections.  First we give some general definitions.

A function, $\transv{e}$, on expression terms returns the result
of translating an expression $e$, assuming that the expression is {\em not}
a distributed array.

Translation functions for distributed-array-expressions are more complicated.
In section
\ref{sec:restrictedForm}
we defined a subset of {\em constructive} distributed-array-valued expressions.
The remaining {\em non-constructive} distributed-array-valued expressions
are:
\begin{description}
\item{\bf a)}
distributed-array-valued local variable access,
\item{\bf b)}
distributed-array-valued field access,
\item{\bf c)}
conditional expression, in which the second or third operands are
distributed arrays.
\item{\bf d)}
assignment expression, in which the left-hand operand is a
distributed-array-valued variable.
\end{description}

The constructive expressions only appear in restricted contexts and
do not have translation functions in their own right (instead
they are effectively handled as part of the translation of a top-level
assignment statements).  For {\em non-constructive} distributed-array-valued
expressions there are $2 + R$ separate parts of the evaluation:
$\transd{e}$, $\transb{e}$ and $\transr{0}{e}$, \ldots, $\transr{R-1}{e}$,
where $R$ is the rank of the array.  The interpretation of these separate
terms will be given in the following sections.

Finally the translation function for {\em statements}, $\trans{S}{p}$,
translates the statement or block $S$ in the context of $p$ as active
process group.  In the schemas given below for translation of statements
we will just use the name {\em apg} to refer to the effective active
process group.  Hence a schema of the form
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\textbf{SOURCE:} \\
\verb$    $$S$ \\
\\
\textbf{TRANSLATION:} \\
\verb$    $$S'$
\end{tabbing}
\end{bigCode}
\end{displaymath}
should be read more precisely as 
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\textbf{SOURCE:} \\
\verb$    $$s \quad \equiv \quad S$ \\
\\
\textbf{TRANSLATION:} \\
\verb$    $$\trans{s}{\mbox{\it apg}} \quad \equiv \quad S'$
\end{tabbing}
\end{bigCode}
\end{displaymath}


% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage
\verb$ $
\newpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
T \verb$ [[$ \mbox{\it attr}_0 \verb$, $ \ldots \verb$, $ \mbox{\it attr}_{R-1} \verb$]] $ \mbox{\it bras} \verb$ $ a \verb$ = $e\verb$ ;$
\end{displaymath}

\textbf{TRANSLATION:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$T$\verb$ [] $$\mbox{\it bras}$\verb$ $$a'_{\mbox{\small dat}}$\verb$ = $$\transd{e}$\verb$ ;$ \\
\\
\verb$ArrayBase $$a'_{\mbox{\small bas}}$\verb$ = $$\transb{e}$\verb$ ;$ \\
\\
$\mbox{\it DIMENSION\_TYPE}\left(\mbox{\it attr}_0\right)$\verb$ $$a'_0$\verb$ = $$\transr{0}{e}$\verb$ ;$ \\
\verb$...$ \\
$\mbox{\it DIMENSION\_TYPE}\left(\mbox{\it attr}_{R-1}\right)$\verb$ $$a'_{R-1}$\verb$ = $$\transr{R-1}{e}$\verb$ ;$ \\
\end{tabbing}
\end{bigCode}
\end{displaymath}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{$T$ is a Java type,} \\
\mbox{$R$ is the rank of the declared array,} \\
\mbox{each term $\mbox{\it attr}_r$ is a single hyphen, {\tt -}, or a single asterisk, {\tt *},} \\
\mbox{the term $\mbox{\it bras}$ is a string of zero or more bracket pairs, {\tt []},} \\
\mbox{the identifier $a$ is the name of the declared array in the source program,} \\
\mbox{the expression $e$ is an optional initializer in the source program,} \\
\mbox{$a'_{\mbox{\small dat}} = \mbox{\it TRANS\_ID}(a)$,} \\
\mbox{$a'_{\mbox{\small bas}} = \mbox{\it TRANS\_ID\_BAS}(a)$,} \\
\mbox{$a'_r = \mbox{\it TRANS\_ID\_DIM}(a, r)$, and} \\
\mbox{the macros {\it TRANS\_ID}, {\it TRANS\_ID\_BAS}, {\it TRANS\_ID\_DIM} and} \\
\quad \quad \mbox{{\it DIMENSION\_TYPE} are defined in the text.}
\end{array}
\end{displaymath}
\begin{displaymath}
\begin{array}{l}
\end{array}
\end{displaymath}
\caption{Translation of a distributed-array-valued variable declaration.}
\label{fig:transArrayDeclaration}

\end{figure}

\clearpage

\subsection{Translating variable declarations}
\label{sec:transArrayDeclaration}

The general scheme for translating declaration of a field or local variable
holding a distributed array reference is illustrated in Figure
\ref{fig:transArrayDeclaration}.  The single variable in the source
program is converted to $2 + R$ variables in the output program, where
$R$ is the rank of the array.

The macro {\it TRANS\_ID} transforms the identifier of the field in a way
that encodes the dimension signature of the result.  This encoding
is necessary in order that the HPJava type signature of a field can be
reconstructed from the class file of a translated (and compiled) class.
Note that the {\em element type} of the distributed array field is known
from the type of Java array in translated field---this information is
automatically encoded in the Java class file.

The values returned by the {\it TRANS\_ID} macro and the macros
{\it TRANS\_ID\-\_BAS} and {\it TRANS\_ID\_DIM}
all start with the the original $a$ string, followed by the
suffix separator ``{\tt \_\_\$}''.  This is followed by
a string characteristic of the individual macro, described next.

The characteristic string for {\it TRANS\_ID}($a$) is a
string of $R$ letters, each of which is ``\verb$D$'' (for a distributed
dimension) or ``\verb$S$'' (for a sequential dimension).
The characteristic string for {\it TRANS\_ID\_BAS}($a$)
is ``\verb$bas$''. 
The characteristic string for {\it TRA\-NS\_ID\_DIM}($a$, $r$)
is the decimal representation of the constant integer value $r$.

The variable 
$\mbox{\it TRANS\_ID}(a)$
will hold a reference to the
Java array containing the locally held elements of the distributed array.
This is the value that would be returned by the inquiry $a$\verb$.dat()$
in the source program.  

The $\mbox{\it TRANS\_ID\_BAS}(a)$ will hold an
instance of the class {\tt ArrayBase}.
Instances of this class
contain a group object (the value that would be returned by
the inquiry $a$\verb$.grp()$ in the source program)
and an offset from the start of the Java array, where the first locally held
element of the distributed array is stored (the value that would
be returned by the inquiry $a$\verb$.bas()$ in the source program).

The $R$ variables
$\mbox{\it TRANS\_ID\_DIM}(a, 0)$
\ldots, 
$\mbox{\it TRANS\_ID\_DIM}(a, R-1)$
will hold descriptors for the dimensions of the arrays.  
The macro {\it DIMENSION\_\-TYPE} is defined as
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it DIMENSION\_TYPE}\left(\mbox{\it attr}_r\right) \equiv$
    \verb$ArrayDim$
\end{tabbing}
\end{bigCode}
\end{displaymath}
if the term $\mbox{\it attr}_r$ is a hyphen, {\tt -}, or
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it DIMENSION\_TYPE}\left(\mbox{\it attr}_r\right) \equiv$
\verb$SeqArrayDim$
\end{tabbing}
\end{bigCode}
\end{displaymath}
if the term $\mbox{\it attr}_r$ is an asterisk, {\tt *}.
Instances of the class {\tt ArrayDim}
contain a distributed range object and an associated
``memory stride''---the values that would be returned by the inquiries
$a$\verb$.rng($$r$\verb$)$ and $a$\verb$.str($$r$\verb$)$,
respectively, in the source program.  These values are stored in the
fields {\tt range} and {\tt stride} of {\tt ArrayDim}.  The class
{\tt SeqArrayDim} is a subclass of {\tt ArrayDim} that contains extra
information that can be used to simplify the computations associated
with subscripting a sequential array dimension.

If, for example, a class in the source program has a field:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    float [[-,-,*]] bar ;
\end{verbatim}
\end{code}
\end{displaymath}
the translated class can be assumed to have the five fields:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    float [] bar__$DDS ;

    ArrayBase bar__$bas ;

    ArrayDim bar__$0 ;
    ArrayDim bar__$1 ;
    SeqArrayDim bar__$2 ;
\end{verbatim}
\end{code}
\end{displaymath}

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage
\verb$ $
\newpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$T$\verb$ $$f$\verb$($$U_0$\verb$ $$v_0$\verb$, $$\ldots$\verb$, $$U_{N-1}$\verb$ $$v_{N-1}$\verb$) {$$S$\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}

\textbf{TRANSLATION:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$T$\verb$ $$f$\verb$($\=$\mbox{\it TRANS\_PARAMS}(U_0\verb$ $v_0, \ldots, U_{N-1}\verb$ $v_{N-1})$\verb$, Group $$p$\verb$) {$ \\
\verb$    $$\trans{S}{p}$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{$T$ is a Java type,} \\
\mbox{the identifier $f$ is the name of the method,} \\
\mbox{each of the terms $U_0$, \ldots, $U_{N-1}$ is a Java type or a distributed array type,} \\
\mbox{$v_0$, \ldots, $v_{N-1}$ are parameter names appearing in the original program,} \\
\mbox{$S$ is a block of statements in the original program,} \\
\mbox{$\mbox{\it p}$ is the name of a new parameter, and} \\
\mbox{the macro {\it TRANS\_PARAMS} is defined in the text.}
\end{array}
\end{displaymath}
\caption{Translation of method declaration.}
\label{fig:transNonArrayMethodDeclaration}
\end{figure}

\clearpage

\subsection{Translating method declarations}
\label{sec:transMethodDeclaration}

There are two cases to consider.  The case where the result of the
method is not a distributed array, and the case where the result is
a distributed array.  First we consider the case where the result
is {\em not} a distributed array.

The general scheme is illustrated in Figure
\ref{fig:transNonArrayMethodDeclaration}.  This scheme is modified
in trivial ways if the method has a {\tt void} result, or involves other
modifiers (they are copied to the translated code).

The macro {\it TRANS\_PARAMS} evaluates to a list of formal parameters.
We will define it in terms the simpler macro {\it TRANS\_PARAM}
which operates on a single formal parameter declaration:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it TRANS\_PARAMS}\left(U_0\verb$ $v_0, \ldots,
                               U_{N-1}\verb$ $v_{N-1}\right) \equiv$ \\
\verb$        $$\mbox{\it TRANS\_PARAM}\left(U_0\verb$ $v_0\right)$\verb$, $
$\ldots$\verb$, $$\mbox{\it TRANS\_PARAM}\left(U_{N-1}\verb$ $v_{N-1}\right)$
 \\
\end{tabbing}
\end{bigCode}
\end{displaymath}
Now, if $U$ is not a distributed array type, we have
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it TRANS\_PARAM}\left(U\verb$ $v\right) \equiv \quad U$\verb$ $$v$
\end{tabbing}
\end{bigCode}
\end{displaymath}
Otherwise, if $U$ has the form
\begin{displaymath}
T \verb$ [[$ \mbox{\it attr}_0 \verb$, $ \ldots \verb$, $ \mbox{\it attr}_{R-1} \verb$]] $ \mbox{\it bras} \verb$ $ a \verb$ ;$
\end{displaymath}
(where as usual $T$ is a Java type and as usual
each term $\mbox{\it attr}_r$ is a single hyphen, {\tt -}, or
a single asterisk, {\tt *}, and the term $\mbox{\it bras}$ is a string
of zero or more bracket pairs, {\tt []})
then the macro {\it TRANS\_PARAM} is defined by
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it TRANS\_PARAM}\left(U\verb$ $v\right) \equiv$ \\
\verb$        $$T$\verb$ [] $$\mbox{\it bras}$\verb$ $$a'_{\mbox{\small dat}}$\verb$, ArrayBase $$a'_{\mbox{\small bas}}$\verb$,$ \\
\verb$                $$\mbox{\it DIMENSION\_TYPE}\left(\mbox{\it attr}_0\right)$\verb$ $$a'_0$\verb$, $$\ldots$\verb$, $ \\
\verb$                        $$\mbox{\it DIMENSION\_TYPE}\left(\mbox{\it attr}_{R-1}\right)$\verb$ $$a'_{R-1}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $a'_{\mbox{\small dat}}$, $a'_{\mbox{\small bas}}$, 
$a'_{\mbox{\small grp}}$, and $a'_0$, \ldots, $a'_{R-1}$ are new formal
parameter names that may be obtained from $a$ using the prescription
given in Figure \ref{fig:transArrayDeclaration}.
The macro {\it DIMENSION\_TYPE} is defined in section
\ref{sec:transArrayDeclaration}.

In other words, each distributed array parameter is split into
$2 + R$ parameters.

The final parameter $p$ added by the translator will hold the value
of the {\em active process group} in effect at the point of invocation
of the method.

If, for example, a class in the source program has a method:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    void foo(int [[-,-,*]] bar) {...}
\end{verbatim}
\end{code}
\end{displaymath}
the translated class can be assumed to have the method:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    void foo(int [] bar__$DDS, ArrayBase bar__$BAS,
             ArrayDim bar__$0, ArrayDim bar__$1, SeqArrayDim bar__$2,
             Group p) {...}
\end{verbatim}
\end{code}
\end{displaymath}

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$T$\verb$ [[$$\mbox{\it attr}_0$\verb$, $$\ldots$\verb$, $$\mbox{\it attr}_{R-1}$\verb$]] $$\mbox{\it bras}$\verb$ $$f$\verb$($$U_0$\verb$ $$v_0$\verb$, $$\ldots$\verb$, $$U_{N-1}$\verb$ $$v_{N-1}$\verb$) {$ \\
\verb$    $$S$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}

\textbf{TRANSLATION:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$T$\verb$ [] $$\mbox{\it bras}$\verb$ $$\mbox{\it TRANS\_ID}(f)$\verb$($\=\verb$DAD $$d$\verb$,$ \\
\>$\mbox{\it TRANS\_PARAMS}(U_0\verb$ $v_0, \ldots, U_{N-1}\verb$ $v_{N-1})$\verb$,$ \\
\>\verb$Group $$p$\verb$) {$ \\
\verb$    $$\trans{S}{p}$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{$T$ is a Java type,} \\
\mbox{$R$ is the rank of the returned array,} \\
\mbox{each term $\mbox{\it attr}_r$ is a single hyphen, {\tt -}, or a single asterisk, {\tt *},} \\
\mbox{the term $\mbox{\it bras}$ is a string of zero or more bracket pairs, {\tt []},} \\
\mbox{the identifier $f$ is the name of the method,} \\
\mbox{each of the terms $U_0$, \ldots, $U_{N-1}$ is a Java type or a
distributed array type,} \\
\mbox{$v_0$, \ldots, $v_{N-1}$ are parameter names appearing in the
original program,} \\
\mbox{$S$ is a block of statements in the original program,} \\
\mbox{$d$ and $p$ is the names of new parameters, and} \\
\mbox{the macros {\it TRANS\_ID} and {\it TRANS\_PARAMS} is defined in the
text.}
\end{array}
\end{displaymath}
\caption{Translating declaration of method returning a distributed array.}
\label{fig:transArrayMethodDeclaration}
\end{figure}

\clearpage

\subsubsection{Methods that return distributed arrays}

The scheme for translating a distributed-array-valued method declaration
is illustrated in Figure \ref{fig:transArrayMethodDeclaration}.

The macro {\it TRANS\_ID} encodes the dimension signature of the returned
array and is defined in section \ref{sec:transArrayDeclaration}.
The macro {\it TRANS\_PARAMS} was defined earlier in this section.

Instances of the class {\tt DAD} (the initials stand for
{\em Distributed Array Descriptor}) contain a
reference to an {\tt ArrayBase} object, and a vector of $R$ instances
of {\tt ArrayDim}.  These values are stored in fields
{\tt base} and {\tt dimensions}, respectively.
The DAD object will be created in the calling program and passed to
the translated method.
Before the method returns, it will store the parameters
describing the layout of the distributed array result in these fields
(see sections \ref{sec:transArrayMethodInvocation} and
\ref{sec:transReturn}).

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage
\verb$ $
\newpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$T$\verb$($$U_0$\verb$ $$v_0$\verb$, $$\ldots$\verb$, $$U_{N-1}$\verb$ $$v_{N-1}$\verb$) {$$S$\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}

\textbf{TRANSLATION:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$T$\verb$($\=$\mbox{\it TRANS\_PARAMS}(U_0\verb$ $v_0, \ldots, U_{N-1}\verb$ $v_{N-1})$\verb$, Group $$p$\verb$) {$ \\
\verb$    $$\trans{S}{p}$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{$T$ is a Java type, the name of the declaring class,} \\
\mbox{each of the terms $U_0$, \ldots, $U_{N-1}$ is a Java type or a distributed array type,} \\
\mbox{$v_0$, \ldots, $v_{N-1}$ are parameter names appearing in the original program,} \\
\mbox{$S$ is a block of statements in the original program,} \\
\mbox{$\mbox{\it p}$ is the name of a new parameter, and} \\
\mbox{the macro {\it TRANS\_PARAMS} is defined in the text.}
\end{array}
\end{displaymath}
\caption{Translation of constructor declaration.}
\label{fig:transConstructorDeclaration}
\end{figure}

\clearpage

\subsection{Translating constructor declarations}
\label{sec:transConstructorDeclaration}

The scheme for translating declaration of constructors in HPspmd
classes is illustrated in Figure \ref{fig:transConstructorDeclaration}.

The macro {\it TRANS\_PARAMS} was defined in section
\ref{sec:transMethodDeclaration}.


% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage
\newpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$on ($$p$\verb$) $$S$ \\
\end{tabbing}
\end{bigCode}
\end{displaymath}

\textbf{TRANSLATION:}

\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$if ($$p$\verb$.member()) {$ \\
\verb$    $$\trans{S}{p}$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{$p$ is a simple expression in the source, and} \\
\mbox{$S$ is a statement in the source program.}
\end{array}
\end{displaymath}

\caption{Translation of {\tt on} construct.}
\label{fig:transOn}
\end{figure}

\clearpage

\subsection{Translating {\tt on} constructs}

A translation for the {\tt on} construct is given in Figure
\ref{fig:transOn}.  Note that pre-translation will have reduced the
expression $p$ to a simple expression, which itself requires no
translation.

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage
\newpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$at ($$i$\verb$ = $$x$\verb$[$$n$\verb$]) $$S$ \\
\end{tabbing}
\end{bigCode}
\end{displaymath}

\textbf{TRANSLATION:}

\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$Location $$l$\verb$ = $
          $x$\verb$.location($$n$\verb$) ;$ \\
\\
\verb$Dimension $$d$\verb$ = $$x$\verb$.dim() ;$ \\
\verb$if ($$d$\verb$.crd() == $$l$\verb$.crd) {$ \\
\verb$    Group $$p$\verb$ = (Group) $$\mbox{\it apg}$\verb$.clone() ;$ \\
\verb$    $$p$\verb$.restrict($$x$\verb$.dim() ;$ \\
\\
\verb$    $$\trans{S}{p}$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{$i$ is an index name in the source program,} \\
\mbox{$x$ and $n$ are simple expressions in the source program,} \\
\mbox{$S$ is a statement in the source program, and} \\
\mbox{$l$, $d$ and $p$ are the names of new variables.} \\
\end{array}
\end{displaymath}

\caption{Translation of {\tt at} construct.}
\label{fig:transAt}
\end{figure}

\clearpage

\subsection{Translating {\tt at} constructs}
\label{sec:transAt}

A translation for the {\tt at} construct is given in Figure
\ref{fig:transAt}.  Note that pre-translation will have reduced
$x$ and $n$ to simple expressions.

The coordinate and local subscript associated with the specified
location is returned by the method, {\tt location()}, which is a member
of the {\tt Range} class.  It takes one argument, the global subscript,
and returns an object of class {\tt Location}, declared as:
\begin{displaymath}
\begin{code}
\begin{verbatim}
  class Location {
    public int crd ;
    public int sub ;
  }
\end{verbatim}
\end{code}
\end{displaymath}
%The {\tt restrict()} method will throw an exception if $d$ is
%not an effective dimension of $\mbox{\it apg}$.

The {\em local subscript} for the index $i$ is the value of
$l\verb$.sub$$.  This value is used in subscripting distributed arrays.

The {\em global index} for the index $i$ is the value of
$n$.  This value is used in evaluating the global
index expression $i\verb$`$$.  Note that pretranslation has reduced
the expression $n$ to a simple expression that is constant in the body of
the construct (by introducing a temporary if necessary).

The {\em shift step} for the index $i$ is defined to be the value
of $x\verb$.str()$$.  This value is used in computation of offsets
associated with shifted index subscripts.

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$overall ($$i$\verb$ = $$x$\verb$ for $
                       $\elo$\verb$ : $$\ehi$\verb$ : $$\estp$\verb$) $$S$ \\
\end{tabbing}
\end{bigCode}
\end{displaymath}

\textbf{TRANSLATION:}

\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$Block $$b$\verb$ = $
          $x$\verb$.localBlock($$\transv{\elo}$\verb$, $
                                $\transv{\ehi}$\verb$, $
                                $\transv{\estp}$\verb$) ;$ \\
\\
\verb$Group $$p$\verb$ = (Group) $$\mbox{\it apg}$\verb$.clone() ;$ \\
$p$\verb$.restrict($$x$\verb$.dim()) ;$ \\
\\
\verb$for (int $$l$\verb$ = 0 ; $$l$\verb$ < $$b$\verb$.count ; $$l$\verb$++) {$ \\
\verb$    int $$\mbox{\it sub}$\verb$ = $$b$\verb$.sub_bas + $$b$\verb$.sub_stp * $$l$\verb$ ;$ \\
\verb$    int $$\mbox{\it glb}$\verb$ = $$b$\verb$.glb_bas + $$b$\verb$.glb_stp * $$l$\verb$ ;$ \\
\\
\verb$    $$\trans{S}{p}$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{$i$ is an index name in the source program,} \\
\mbox{$x$ is a simple expression in the source program,} \\
\mbox{$\elo$, $\ehi$, and $\estp$ are expressions in the source,} \\
\mbox{$S$ is a statement in the source program, and} \\
\mbox{$b$, $p$, $l$, $\mbox{\it sub}$ and $\mbox{\it glb}$
      are names of new variables.} \\
\end{array}
\end{displaymath}

\caption{Translation of {\tt overall} construct.}
\label{fig:transOverall}
\end{figure}

\clearpage

\subsection{Translating {\tt overall} constructs}
\label{sec:transOverall}

A translation for the {\tt overall} construct is given in Figure
\ref{fig:transOverall}.  The {\tt localBlock()} method and
the {\tt Block} class have been discussed at length in section
\ref{sec:blockParameters}.

The {\em local subscript} for the index $i$ is the value of
$\mbox{\it sub}$.  This value is used in subscripting distributed
arrays.

The {\em global index} for the index $i$ is the value of
$\mbox{\it glb}$.  This value is used in evaluating the global
index expression $i\verb$`$$.

The {\em shift step} for the index $i$ is defined to be the value
of $x\verb$.str()$$.  This value is used in computation of offsets
associated with shifted index subscripts.

%The {\tt restrict()} method will throw an exception if $x\verb$.dim()$$ is
%not an effective dimension of $\mbox{\it apg}$.

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & i\verb$`$
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transv{e} & \equiv & \mbox{\it glb}
\end{eqnarray*}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{$i$ is an index name in the source program, and} \\
\mbox{{\it glb} is the global index variable for the index $i$.}
\end{array}
\end{displaymath}
\caption{Translation of global index for $i$.}
\label{fig:transGlobalIndex}
\end{figure}

\clearpage

\subsection{Translating global index expression}

The scheme is illustrated in Figure \ref{fig:transGlobalIndex}.
The global index variable associated with a distributed index is defined
in sections \ref{sec:transAt} and \ref{sec:transOverall}.

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & a
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transd{e} & \equiv & a'_{\mbox{\small dat}} \\
\\
\transb{e} & \equiv & a'_{\mbox{\small bas}} \\
\\
\transr{0}{e}   & \equiv & a'_0 \\
\ldots \\
\transr{R-1}{e} & \equiv & a'_{R-1}
\end{eqnarray*}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{$a$ is an array name in the source program, rank $R$, and} \\
\mbox{$a'_{\mbox{\small dat}}$, $a'_{\mbox{\small bas}}$, 
and $a'_0, \ldots, a'_{R-1}$ are corresponding} \\
\quad \quad \mbox{names of variables in the translated program.} \\
\end{array}
\end{displaymath}
\caption{Translation of a distributed-array-valued variable access.}
\label{fig:transLocalVariableAccess}
\end{figure}

\clearpage

\subsection{Translating variable accesses}

We only need to consider specially the case where the variable is
a distributed array.  The general scheme is illustrated in Figure
\ref{fig:transLocalVariableAccess}.  This applies to the case of a
variable that is a {\em simple identifier}---a local variable, method
or constructor argument, or instance variable of the current object.
The case of a field reference with an object prefix will be covered
in section \ref{sec:transFieldAccesses}.

The names $a'_{\mbox{\small dat}}$, $a'_{\mbox{\small bas}}$,
and $a'_0, \ldots, a'_{R-1}$ are the names
introduced by the translator when translating the corresponding
field, local variable declaration or formal parameter in the source
program (see sections \ref{sec:transArrayDeclaration} and
\ref{sec:transMethodDeclaration}).

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & e_t \verb$ = $ e_s
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transd{e} & \equiv & \transd{e_t} \verb$ = $ \transd{e_s} \\
\transb{e} & \equiv & \transb{e_t} \verb$ = $ \transb{e_s} \\
\transr{0}{e}   & \equiv & \transr{0}{e_t} \verb$ = $ \transr{0}{e_s} \\
\ldots \\
\transr{R-1}{e} & \equiv & \transr{R-1}{e_t} \verb$ = $ \transr{R-1}{e_s}
\end{eqnarray*}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{$e_t$ has distributed array type,} \\
\mbox{$e_s$ is assignment convertible to the type of $e_t$, and} \\
\mbox{$R$ is the rank of both arrays.} \\
\end{array}
\end{displaymath}
\caption{Translation of a distributed-array assignment.}
\label{fig:transAssignment}
\end{figure}

\clearpage

\subsection{Translating assignment expressions}

When the expresssions involved are distributed arrays, the general scheme
is illustrated in Figure \ref{fig:transAssignment}.
This translation applies to assignments where the right-hand-side of
the assignment is {\em not} a constructive expression.  Assignments
involving constructive expressions are handled in the following sections.

{\em [For completeness, also need to add translation of assignment
expressions with non-constructive, array-valued right-hand-sides,
when they appear as {\em statement expressions}.  These can be top-level
assignments or they can appear in the header of {\tt for} constructs:
both these cases need to be covered, though in both cases the translation
is just the sequence of Java assignment statements produced by the array
translation functions.]}

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$a$ \verb$ = new $$T$\verb$ [[$$e_0$\verb$, $ \ldots
                      \verb$, $$e_{R-1}$\verb$]] $$\mbox{\it bras}$\verb$ on $$p$\verb$ ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}

\textbf{TRANSLATION:}

\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$s$ \verb$ = 1 ;$ \\
$b$ \verb$ = 0 ;$ \\
$\mbox{\it DEFINE\_DIMENSION}\left(\transr{R-1}{a}, e_{R-1}, s, b\right)$ \\
\verb$...$ \\
$\mbox{\it DEFINE\_DIMENSION}\left(\transr{0}{a}, e_0, s, b\right)$ \\
$\transd{a}$\verb$ = $$p$\verb$.member() ? new $$T$\verb$ [$$s$\verb$] $$\mbox{\it bras}$\verb$ : null ;$ \\
$\transb{a}$\verb$ = new ArrayBase($$p$\verb$, $$b$\verb$) ;$ \\
\end{tabbing}
\end{bigCode}
\end{displaymath}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{$T$ is a Java type,} \\
\mbox{$R$ is the rank of the created array,} \\
\mbox{each $e_r$ is either a range-valued or an integer-valued simple expression} \\
\quad \quad \mbox{in the source program,} \\
\mbox{the term $\mbox{\it bras}$ is a string of zero or more bracket pairs, {\tt []},} \\
\mbox{$p$ is a simple expression in the source program,} \\
\mbox{the expression $a$ is the assigned array variable in the source program,} \\
\mbox{$s$ and $b$ are names of new temporaries, and} \\
\mbox{the macro {\it DEFINE\_DIMENSION} is defined in the text.}
\end{array}
\end{displaymath}

\caption{Translation of distributed array creation expression.}
\label{fig:transArrayCreation}
\end{figure}

\clearpage

\subsection{Translating distributed array creation}

The pre-translator ensures that distributed array creation only
appears on the right-hand-side of a top-level assignment, so we
only need to consider that case.
The scheme
is illustrated in Figure \ref{fig:transArrayCreation}.
If the ``$\verb$on $p$'' clause is omitted in the source program,
the value of $p$ in the translation can be taken to be
{\it apg}.  Since $\mbox{\it apg}$\verb$.member()$ is always true, 
the computation of $\transd{a}$ can then be simplified to just
\verb$new $$T$\verb$ [$$s$\verb$]$\verb$ $$\mbox{\it bras}$.

The macro {\it DEFINE\_DIMENSION} is defined as follows:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it DEFINE\_DIMENSION}\left(a'_r, e_r, s, b\right) \equiv$ \\
\verb$            $$a'_r$\verb$ = $$e_r$\verb$.arrayDim($$s$\verb$) ;$ \\
\verb$            $$b$\verb$ += $$s$\verb$ * $$e_r$\verb$.loExtension() ;$ \\
\verb$            $$s$\verb$ *= $$e_r$\verb$.volume() ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}
if the expression $e_r$ is a range, or
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it DEFINE\_DIMENSION}\left(a'_r, e_r, s, b\right) \equiv$ \\
\verb$            $$a'_r$\verb$ = new SeqArrayDim($$e_r$\verb$, $$s$\verb$) ;$ \\
\verb$            $$s$\verb$ *= $$e_r$\verb$ ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}
if the expression $e_r$ is an integer.  As each dimension is processed,
the memory stride for the next dimension is computed by multiplying the
variable $s$ by the number of locally held range elements in the current
dimension.  The final value of $s$ is the total number of locally held
elements.  The variable $b$ is incremented to allow space for a lower
ghost regions, below the base of the physical array, if this is demanded
by the ranges involved.

The method {\tt arrayDim()} on the {\tt Range} class creates an instance
of {\tt ArrayDim}, with the memory stride specified in its argument.  It is
used in place of a call to the {\tt ArrayDim} constructor because {\tt
arrayDim()} has the property that if the range is actually a collapsed
range, the returned object will be an instance of the {\tt
SeqArrayDim} subclass.  This allows a new array created with a collapsed
range to be cast to an array with a sequential dimension, should it
prove necessary at a later stage (see section \ref{sec:transCasts}).

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & a \verb$ [$ e_0 \verb$, $ \ldots \verb$, $ e_{R-1} \verb$]$
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transv{e} & \equiv & \transd{a}\verb$ [$\mbox{\it OFFSET}(a, e_0, \ldots, e_{R-1})\verb$]$
\end{eqnarray*}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{The expression $a$ is the subscripted array,} \\
\mbox{each term $e_r$ is either an integer, a distributed index name,} \\
\quad \quad \mbox{or a shifted index expression, and} \\
\mbox{the macro {\it OFFSET} is defined in the text.}
\end{array}
\end{displaymath}
\caption{Translation of array access expression.}
\label{fig:transArrayAccess}
\end{figure}

\clearpage

\subsection{Translating element access}
\label{sec:translatingAccess}

We only need to consider the case where the array reference is a distributed
array.
The general scheme is illustrated in Figure \ref{fig:transArrayAccess}.
The macro {\it OFFSET} is defined as
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it OFFSET}\left(a, e_0, \ldots, e_{R-1}\right) \equiv$ \\
\\
\verb$    $$\transb{a}$\verb$.base$
    \=\verb$ + $\=$\mbox{\it OFFSET\_DIM}\left(\transr{0}{a}, e_0\right)$ \\
    \>          \>\dots \\
    \>\verb$ + $$\mbox{\it OFFSET\_DIM}\left(\transr{R-1}{a}, e_{R-1}\right)$
\end{tabbing}
\end{bigCode}
\end{displaymath}
There are three cases for the macro {\it OFFSET\_DIM} depending on
whether the subscript argument is a distributed index, a shifted index,
or an integer subscripts (in a sequential dimension).

If $e_r$ is a distributed index $i$, then
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it OFFSET\_DIM}\left(a'_r, e_r\right) \equiv \quad a'_r$\verb$.stride * $$\mbox{\it sub}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $\mbox{\it sub}$ is the local subscript variable for
this index (see sections \ref{sec:transAt}
and \ref{sec:transOverall}).  Otherwise if
$e_r$ is a shifted index $i \pm d$, then
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it OFFSET\_DIM}\left(a'_r, e_r\right) \equiv \quad a'_r$\verb$.stride * ($$\mbox{\it sub}$\verb$ $$\pm$\verb$ $$\mbox{\it shf\_stp}$\verb$ * $$\transv{d}$\verb$)$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $\mbox{\it sub}$ is the local
subscript variable and $\mbox{\it shf\_stp}$ is the shift step for $i$
(again, see sections \ref{sec:transAt} and \ref{sec:transOverall}).
Otherwise if $e_r$ is an integer expression (which implies
that $a'_r$ has type {\tt SeqArrayDim}), then
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it OFFSET\_DIM}\left(a'_r, e_r\right) \equiv \quad a'_r$\verb$.off_bas + $$a'_r$\verb$.off_stp * $$\transv{e_r}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
The fields {\tt off\_bas} and {\tt off\_stp} are initialized
by the constructors for {\tt SeqArray\-Dim}.  They do not exist
in the superclass {\tt ArrayDim}.

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]

\textbf{SOURCE:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$v$\verb$ = $$a$\verb$ [[$$\mbox{\it subs}_0$\verb$, $$\ldots$\verb$, $ 
                          $\mbox{\it subs}_{R-1}$\verb$]] ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}

\textbf{TRANSLATION:}

\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, 0, a, 0\right)$ \\
$\transd{v}$\verb$ = $$\transd{a}$\verb$ ;$ \\
$\transb{v}$\verb$ = $$\transb{a}$\verb$ ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{The expression $v$ is the assigned array variable in the source program,} \\
\mbox{the simple expression $a$ is the subscripted array in the source program,} \\
\mbox{each term $\mbox{\it subs}_s$ is either an integer, a triplet, or {\tt <>},} \\
\mbox{$b$ is the name of a new temporary, and} \\
\mbox{the macro {\it PROCESS\_SUBSCRIPTS} is defined in the text.}
\end{array}
\end{displaymath}

\caption{Translation of array section with no scalar subscripts.}
\label{fig:transArraySectionSimple}
\end{figure}

\clearpage

\subsection{Translating array sections}
\label{sec:transArraySections}

The rules for translating array sections are more complicated than
any other part of the basic translation scheme.

We will break it down into three cases: the case where there are no
scalar subscripts---integer or distributed index; the case where
integer scalar subscripts appear in {\em sequential}
dimensions only; and the general case where scalar subscripts may
appear in distributed dimensions.  The scheme for translating the first
case is illustrated in Figure
\ref{fig:transArraySectionSimple}.

The macro {\it PROCESS\_SUBSCRIPTS} will be defined here in a
tail-recursive way.  The intention is that it should be expanded to a
compile-time loop over $s$.

Let $R$ be the rank of the subscripted array.
If $s = R$, then the macro
$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, r, a, s\right)$
is empty.
Otherwise, if $\mbox{\it subs}_s$ is the degenerate triplet, {\tt :}, then
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, r, a, s\right) \equiv$ \\
\verb$            $$\transr{r}{v}$\verb$ = $$\transr{s}{a}$\verb$ ;$ \\
\verb$            $$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, r+1, a, s+1\right)$
\end{tabbing}
\end{bigCode}
\end{displaymath}
Otherwise, if $\mbox{\it subs}_s$ is the triplet,
$\elo$\verb$:$$\ehi$\verb$:$$\estp$, and the $s$th dimension of 
$a$ is distributed, then
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, r, a, s\right) \equiv$ \\
\verb$    $$\transr{r}{v}$\verb$ = $$a'_s$\verb$.range.subrng($$\elo'$\verb$, $$\ehi'$\verb$, $$\estp'$\verb$).arrayDim($$a'_s$\verb$.stride) ;$ \\
\verb$    $$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, r+1, a, s+1\right)$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $a'_s = \transr{s}{a}$,
$\elo' = \transv{\elo}$, $\ehi' = \transv{\ehi}$, and $\estp' = \transv{\estp}$.
Otherwise, if $\mbox{\it subs}_s$ is the triplet,
$\elo$\verb$:$$\ehi$\verb$:$$\estp$, and the $s$th dimension of 
$a$ is sequential, then
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, r, a, s\right) \equiv$ \\
\verb$  $$\transr{r}{v}$\verb$ = new SeqArrayDim($$a'_s$\verb$.range.subrng($$\elo'$\verb$, $$\ehi'$\verb$, $$\estp'$\verb$), $$a'_s$\verb$.stride) ;$ \\
\verb$  $$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, r+1, a, s+1\right)$
\end{tabbing}
\end{bigCode}
\end{displaymath}
with definitions as above.  Two similar cases using the two-argument
form of {\tt subrng()} take care of triplets of the form $\elo$\verb$:$$\ehi$.
\enlargethispage*{1000pt}
Otherwise, if $\mbox{\it subs}_s$ is the splitting subscript, \verb$<>$, then
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, r, a, s\right) \equiv$ \\

\verb$        $$x$\verb$ = $$a'_s$\verb$.range ;$ \\
\verb$        $$u$\verb$ = $$a'_s$\verb$.stride ;$ \\
\verb$        $$z$\verb$ = $$x$\verb$.shell() ;$ \\
\verb$        $$\transr{r+1}{v}$\verb$ = new SeqArrayDim($$z$\verb$, $$u$\verb$) ;$ \\
\verb$        $$\transr{r}{v}$\verb$ = new ArrayDim($$x$\verb$.dim(), $$u$\verb$ * $$z$\verb$.volume()) ;$ \\
\verb$        $$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, r+2, a, s+1\right)$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $x$, $u$, and $z$ are the names of new temporaries.  
%Use of the inquiry {\tt dim()} should be replaced by the inquiry {\tt
%kernel()} if we desire to support block-cyclic ranges.

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]

\textbf{SOURCE:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$v$\verb$ = $$a$\verb$ [[$$\mbox{\it subs}_0$\verb$, $$\ldots$\verb$, $ 
                          $\mbox{\it subs}_{R-1}$\verb$]] ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}

\textbf{TRANSLATION:}

\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$b$\verb$ = $$\transb{a}$\verb$.base ;$ \\
$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, 0, a, 0\right)$ \\
$\transd{v}$\verb$ = $$\transd{a}$ \\
$\transb{v}$\verb$ = new ArrayBase($$\transb{a}$\verb$.group, $$b$\verb$) ;$ \\
\end{tabbing}
\end{bigCode}
\end{displaymath}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{The expression $v$ is the assigned array variable in the source program,} \\
\mbox{the simple expression $a$ is the subscripted array in the source program,} \\
\mbox{each term $\mbox{\it subs}_s$ is either an integer, a triplet, or {\tt <>},} \\
\mbox{$b$ is the name of a new temporary, and} \\
\mbox{the macro {\it PROCESS\_SUBSCRIPTS} is defined in the text.}
\end{array}
\end{displaymath}

\caption{Translation of array section without any scalar subscripts in
{\em distributed} dimensions.}
\label{fig:transArraySectionSimple}
\end{figure}

\clearpage

\subsubsection{Integer subscripts in sequential dimensions}

To handle the case where integer subscripts may appear in sequential
dimensions (Figure \ref{fig:transArraySectionSimple}) we add one new
case for the definition of the macro {\it PROCESS\_SUB\-SCRIPTS}.

If $\mbox{\it subs}_s$ is an integer expression, and the $s$th
dimension of $a$ is sequential, then
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, r, a, s\right) \equiv$ \\
\verb$    $$b$\verb$ += $$\mbox{\it OFFSET\_DIM}\left(\transr{s}{a},
                                        \mbox{\it subs}_s\right)$\verb$ ;$ \\
\verb$    $$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, r, a, s+1\right)$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where the macro {\it OFFSET\_DIM} is defined in section 
\ref{sec:translatingAccess}.

{\em [Actually distributed index subscripts in sequential dimensions
could also go here, although it is an odd case.]}

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage
%\verb$ $
%\newpage

\begin{figure}[tbp]

\textbf{SOURCE:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$v$\verb$ = $$a$\verb$ [[$$\mbox{\it subs}_0$\verb$, $$\ldots$\verb$, $ 
                          $\mbox{\it subs}_{R-1}$\verb$]] ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}

\textbf{TRANSLATION:}

\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$b$\verb$ = $$\transb{a}$\verb$.base ;$ \\
$p$\verb$ = (Group) $$\transb{a}$\verb$.group.clone() ;$ \\
$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, 0, a, 0\right)$ \\
$\transd{v}$\verb$ = $$p$\verb$.member() ? $$\transd{a}$\verb$ : null ;$ \\
$\transb{v}$\verb$ = new ArrayBase($$p$\verb$, $$b$\verb$) ;$ \\
\end{tabbing}
\end{bigCode}
\end{displaymath}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{The expression $v$ is the assigned array variable in the source program,} \\
\mbox{the simple expression $a$ is the subscripted array in the source program,} \\
\mbox{each term $\mbox{\it subs}_s$ is either an integer, a triplet, or {\tt <>},} \\
\mbox{$b$ and $p$ are the names of new temporaries, and} \\
\mbox{the macro {\it PROCESS\_SUBSCRIPTS} is defined in the text.}
\end{array}
\end{displaymath}

\caption{Translation of array section allowing scalar subscripts in
distributed dimensions.}
\label{fig:transArraySectionGeneral}
\end{figure}

\clearpage

\subsubsection{Scalar subscripts in distributed dimensions}

The scheme for translating array sections when scalar subscripts
appear in some distributed dimension is illustrated in Figure
\ref{fig:transArraySectionGeneral}.

We add two new cases for the definition of the macro
{\it PROCESS\_SUB\-SCRIPTS}.
If $\mbox{\it subs}_s$ is the integer expression $n$, and
the $s$th dimension of $a$ is distributed, then
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, r, a, s\right) \equiv$ \\
\verb$    $$x$\verb$ = $$a'_s$\verb$.range ;$ \\
\verb$    $$l$\verb$ = $$x$\verb$.location($$n'$\verb$) ;$ \\
\verb$    $$b$\verb$ += $$l$\verb$.sub * $$a'_s$\verb$.stride ;$ \\
\verb$    $$p$\verb$.restrict($$x$\verb$.dim(), $$l$\verb$.crd) ;$ \\
\verb$    $$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, r, a, s+1\right)$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $x$ and $l$ are the names of new temporaries,
$a'_s = \transr{s}{a}$, and $n' = \transv{n}$.
Otherwise, if $\mbox{\it subs}_s$ is a distributed index $i$
or a shifted index $i \pm d$, then
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, r, a, s\right) \equiv$ \\
\verb$    $$b$\verb$ += $$\mbox{\it OFFSET\_DIM}\left(\transr{s}{a},
                                        \mbox{\it subs}_s\right)$\verb$ ;$ \\
\verb$    $$p$\verb$.restrict($$x$\verb$.dim()) ;$ \\
\verb$    $$\mbox{\it PROCESS\_SUBSCRIPTS}\left(v, r, a, s+1\right)$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where in this case $x$ is the range associated with $i$ and
the macro {\it OFFSET\_DIM} is defined in section 
\ref{sec:translatingAccess}.

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]

\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & \eobj\verb$.$a
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transd{e} & \equiv & \eobj\verb$.$a'_{\mbox{\small dat}} \\
\\
\transb{e} & \equiv & \eobj\verb$.$a'_{\mbox{\small bas}} \\
\\
\transr{0}{e}   & \equiv & \eobj\verb$.$a'_0 \\
\ldots \\
\transr{R-1}{e} & \equiv & \eobj\verb$.$a'_{R-1}
\end{eqnarray*}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{the simple expression $\eobj$ has class type,} \\
\mbox{$R$ is the rank of the distributed-array-valued field,} \\
\mbox{the identifier $a$ is the name of the field in the source program, and} \\
\mbox{$a'_{\mbox{\small dat}}$, $a'_{\mbox{\small bas}}$, 
and $a'_0, \ldots, a'_{R-1}$ are corresponding names of fields} \\
\quad \quad \mbox{in the translated program.} \\
\end{array}
\end{displaymath}
\caption{Translation of a distributed-array-valued field access.}
\label{fig:transFieldAccess}
\end{figure}

\clearpage

\subsection{Translating field accesses}
\label{sec:transFieldAccesses}

We only need to consider the case where the field is a distributed array.
The general scheme is illustrated in Figure \ref{fig:transFieldAccess}.

The names $a'_{\mbox{\small dat}}$, $a'_{\mbox{\small bas}}$,
and $a'_0, \ldots, a'_{R-1}$ are the names
introduced by the translator when translating the corresponding field
in the source program (see section \ref{sec:transArrayDeclaration}).

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & \eobj \verb$.$f\verb$($e_0 \verb$, $ \ldots  \verb$, $ e_{N-1}\verb$)$
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transv{e} & \equiv & \transv{\eobj}\verb$.$f\verb$($\mbox{\it TRANS\_ARGS}(e_0, \ldots, e_{N-1})\verb$, $\mbox{\it apg}\verb$)$
\end{eqnarray*}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{The expression $\eobj$ has class type,} \\
\mbox{the identifier $f$ is the name of the method,} \\
\mbox{each term $e_r$ is an actual argument in the original program,} \\
\mbox{the macro {\it TRANS\_ARGS} is defined in the text.}
\end{array}
\end{displaymath}
\caption{Translation of method invocation expression,
where return value is not distributed-array-valued.}
\label{fig:transNonArrayMethodInvocation}
\end{figure}

\clearpage

\subsection{Translating method invocations}
\label{sec:transMethodInvocation}

There are two cases to consider.  The case where the result of the
method is not a distributed array, and the case where the result is
a distributed array.  First we consider the case where the result
is not a distributed array.
The general scheme is illustrated in Figure
\ref{fig:transNonArrayMethodInvocation}.  This scheme is modified
in trivial ways if the method is static, or is applied to the
current object.

The macro {\it TRANS\_ARGS} evaluates to a list of translated expressions.
We will define it in terms of the simpler macro {\it TRANS\_ARG}
which operates on a single argument:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it TRANS\_ARGS}\left(e_0, \ldots, e_{N-1}\right) \equiv$ \\
\verb$         $$\quad \mbox{\it TRANS\_ARG}\left(e_0\right)$\verb$, $
           $\ldots$\verb$, $ $\mbox{\it TRANS\_ARG}\left(e_{N-1}\right)$
\end{tabbing}
\end{bigCode}
\end{displaymath}
Now, if $e$ is not a distributed array, then
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it TRANS\_ARG}\left(e\right) \equiv \quad \transv{e}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
Otherwise, if $e$ 
is a distributed array expression of rank $R$, then
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it TRANS\_ARG}\left(e\right) \equiv$ \\
\verb$        $$\transd{e}$\verb$, $$\transb{e}$\verb$, $$\transr{0}{e}$\verb$, $$\ldots$\verb$, $$\transr{R-1}{e}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
In other words, each distributed array argument is split into $2 +
R$ arguments.

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$v$ \verb$ = $$\eobj$\verb$.$$\mbox{\it id}$\verb$($$e_0$\verb$, $$\ldots$\verb$, $$e_{N-1}$\verb$) ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}

\textbf{TRANSLATION:}

\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$d$\verb$ = new DAD($$R$\verb$) ;$ \\
$\transd{v}$\verb$ = $$\transv{\eobj}$\verb$.$$\mbox{\it TRANS\_ID}(f)$\verb$($$d$\verb$, $$\mbox{\it TRANS\_ARGS}(e_0, \ldots, e_{N-1})$\verb$, $$\mbox{\it apg}$\verb$) ;$ \\
$\transb{v}$\verb$ = $$d$\verb$.base$ \\
$\transr{0}{v}$\verb$ = $$\mbox{\it TRANS\_DIM}(d, 0)$ \\
\verb$...$ \\
$\transr{R-1}{v}$\verb$ = $$\mbox{\it TRANS\_DIM}(d, R-1)$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{The expression $\eobj$ has class type or is a class,} \\
\mbox{the identifier $f$ is the name of the method,} \\
\mbox{each term $e_r$ is an actual argument in the source program,} \\
\mbox{$d$ is a new temporary,} \\
\mbox{$R$ is the rank of the result, and} \\
\mbox{the macros {\it TRANS\_ID}, {\it TRANS\_ARGS}, and {\it TRANS\_DIM}} \\
\quad \quad \mbox{are defined in the text.}
\end{array}
\end{displaymath}
\caption{Translation of array-valued method invocation expression.}
\label{fig:transArrayMethodInvocation}
\end{figure}

\clearpage

\subsubsection{Methods that return distributed arrays}
\label{sec:transArrayMethodInvocation}

The scheme for translating a distributed-array-valued method invocation
is illustrated in Figure \ref{fig:transArrayMethodInvocation}.

The macro {\it TRANS\_ID} encodes the dimension signature of the returned
array and is defined in section \ref{sec:transArrayDeclaration}.
The macro {\it TRANS\_ARGS} was defined earlier in this section.

The macro {\it TRANS\_DIM} is defined as
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it TRANS\_DIM}(d, r) \equiv \quad d$\verb$.dimensions [$$r$\verb$]$
\end{tabbing}
\end{bigCode}
\end{displaymath}
if the $r$th dimension of the result is distributed, or
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it TRANS\_DIM}(d, r) \equiv \quad$ 
\verb$(SeqArrayDim) $$d$\verb$.dimensions [$$r$\verb$]$
\end{tabbing}
\end{bigCode}
\end{displaymath}
if the $r$th dimension is sequential.

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
\verb$return $a\verb$ ;$
\end{displaymath}

\textbf{TRANSLATION:}

\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$d$\verb$.base  = $$\transb{a}$\verb$ ;$ \\
$d$\verb$.dimensions [$$0$\verb$] = $$\transr{0}{a}$\verb$ ;$ \\
\ldots \\
$d$\verb$.dimensions [$$R-1$\verb$] = $$\transr{R-1}{a}$\verb$ ;$ \\
\\
\verb$return $$\transd{a}$\verb$ ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where:
\begin{displaymath}
\begin{array}{l}

\mbox{$a$ is the array-valued result expression,} \\
\mbox{$R$ is its rank, and} \\
\mbox{$d$ is the DAD passed as first argument of the translated method.}
\end{array}
\end{displaymath}
\caption{Translation of return statement in array-valued method.}
\label{fig:transArrayReturn}
\end{figure}

\clearpage

\subsubsection{Translation of {\tt return} statement}
\label{sec:transReturn}

The scheme for translating a {\tt return} statement in the
definition of an array-valued method is illustrated in Figure
\ref{fig:transArrayReturn}.

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & \verb$new $ T \verb$($e_0 \verb$, $ \ldots  \verb$, $ e_{N-1}\verb$)$
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transv{e} & \equiv & \verb$new $ T \verb$($\mbox{\it TRANS\_ARGS}(e_0, \ldots, e_{N-1})\verb$, $\mbox{\it apg}\verb$)$
\end{eqnarray*}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{$T$ is a Java class type,} \\
\mbox{each term $e_r$ is an actual argument in the original program,} \\
\mbox{the macro {\it TRANS\_ARGS} is defined in the text.}
\end{array}
\end{displaymath}
\caption{Translation of class instance creation expression.}
\label{fig:transConstructorInvocation}
\end{figure}

\clearpage

\subsection{Translating constructor invocations}

The rules for translating constructor invocations follow
directly from the rules for method invocations given
in section \ref{sec:transMethodInvocation}.

Figure \ref{fig:transConstructorInvocation} illustrates
the translation for a class instance creation expression.
Explicit constructor invocations (specifying {\tt this} or
{\tt super}) do not introduce any new features%
\footnote{But see the remarks concerning explicit constructor
invocations in section \ref{sec:restrictedForm}.}%
.


% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]

\textbf{SOURCE:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$v$\verb$ = $$a$\verb$ / $$\eloc$\verb$ ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}

\textbf{TRANSLATION:}

\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it RESTRICT\_GROUP}\left(p, \transb{a}\verb$.group$,
                                    \eloc\right)$\verb$ ;$ \\
$\transd{v}$\verb$ = $$p$\verb$.member() ? $$\transd{a}$\verb$ : null ;$ \\
$\transb{v}$\verb$ = new ArrayBase($$p$\verb$, $$\transb{a}$\verb$.base) ;$ \\
$\transr{0}{v}$\verb$ = $$\transr{0}{a}$\verb$ ;$ \\
$\ldots$ \\
$\transr{R-1}{v}$\verb$ = $$\transr{R-1}{a}$\verb$ ;$ \\
\end{tabbing}
\end{bigCode}
\end{displaymath}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{The expression $v$ is the assigned array variable in the source program,} \\
\mbox{the simple expression $a$ is the array in the source program to be restricted,} \\
\mbox{$R$ is the rank of $a$,} \\
\mbox{the expression $\eloc$ is either a distributed index, a shifted index,} \\
\quad \quad \mbox{or a range element,} \\
\mbox{$p$ is a new temporary, and} \\
\mbox{the macro {\it RESTRICT\_GROUP} is defined in the text.}
\end{array}
\end{displaymath}
\caption{Translation of array restriction operation.}
\label{fig:transArrayRestriction}
\end{figure}

\clearpage

\subsection{Translating array restriction}
\label{sec:transArrayRestriction}

The scheme is illustrated in Figure \ref{fig:transArrayRestriction}.

If $\eloc$ is a range element of the form $\erng \verb$[$n\verb$]$$,
the macro {\it RESTRICT\_GROUP} is defined as
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it RESTRICT\_GROUP}\left(p, \egrp, \eloc\right) \equiv$ \\
\verb$    $$x$\verb$ = $$\erng'$\verb$ ;$ \\
\verb$    $$l$\verb$ = $$x$\verb$.location($$n'$\verb$) ;$ \\
\verb$    $$p$\verb$ = (Group) $$\egrp$\verb$.clone() ;$ \\
\verb$    $$p$\verb$.restrict($$x$\verb$.dim(), $$l$\verb$.crd) ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $x$ and $l$ are the names of new temporaries,
$\erng' = \transrV{s}{\erng}$, and $n' = \transv{n}$.
Otherwise, if $\eloc$ is a distributed index $i$ or a shifted index $i \pm d$,
it is defined as
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it RESTRICT\_GROUP}\left(p, \egrp, \eloc\right) \equiv$ \\
\verb$    $$p$\verb$ = (Group) $$\egrp$\verb$.clone() ;$ \\
\verb$    $$p$\verb$.restrict($$x$\verb$.dim()) ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where in this case $x$ is the range associated with $i$.

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$v$\verb$ = $$p$\verb$ / $$\eloc$\verb$ ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}

\textbf{TRANSLATION:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it RESTRICT\_GROUP\_ASSIGN}\left(v, \transv{p}, \eloc\right)$\verb$ ;$ \\
\end{tabbing}
\end{bigCode}
\end{displaymath}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{The expression $v$ is a variable in the source program,} \\
\mbox{the expression $p$ is the group to be restricted,} \\
\mbox{the expression $\eloc$ is either a distributed index, a shifted index,} \\
\quad \quad \mbox{or a range element, and} \\
\mbox{the macro {\it RESTRICT\_GROUP} is defined in the text.}
\end{array}
\end{displaymath}
\caption{Translation of group restriction operation.}
\label{fig:transGroupRestriction}
\end{figure}

\clearpage

\subsection{Translating group restriction}

The scheme is illustrated in Figure \ref{fig:transGroupRestriction}.
If $v$ is a simple expression,
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it RESTRICT\_GROUP\_ASSIGN}\left(v, \egrp, \eloc\right) \equiv$ \\
\verb$    $$\mbox{\it RESTRICT\_GROUP}\left(v, \egrp, \eloc\right)$
\end{tabbing}
\end{bigCode}
\end{displaymath}
Otherwise
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it RESTRICT\_GROUP\_ASSIGN}\left(v, \egrp, \eloc\right) \equiv$ \\
\verb$    $$\mbox{\it RESTRICT\_GROUP}\left(p, \egrp, \eloc\right)$ \\
\verb$    $$v$\verb$ = $$p$\verb$ ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $p$ is the name of a new temporary.
The macro {\it RESTRICT\_GROUP} is defined in section
\ref{sec:transArrayRestriction}.

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & x \verb$ [$ \elo \verb$ : $ \ehi \verb$]$
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transv{e} & \equiv & \transv{x} \verb$.subrng($ \transv{\elo} \verb$, $
                                          \transv{\ehi} \verb$)$ \\
\end{eqnarray*}

\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & x \verb$ [$ \elo \verb$ : $ \ehi \verb$ : $ \estp \verb$]$
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transv{e} & \equiv & \transv{x} \verb$.subrng($ \transv{\elo} \verb$, $
                             \transv{\ehi} \verb$, $ \transv{\estp} \verb$)$ \\
\end{eqnarray*}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{The expression $x$ is the parent range, and} \\
\mbox{$\elo$, $\ehi$ and $\estp$ are integer-valued expressions in the source program.} \\
\end{array}
\end{displaymath}
\caption{Translation of subrange expressions.}
\label{fig:transSubrange}
\end{figure}

\clearpage

\subsection{Translating subrange expressions}

The schemes are illustrated in Figure \ref{fig:transSubrange}.

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & \ebool \verb$ ? $ a \verb$ : $ b
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transd{e} & \equiv & \ebool \verb$ ? $ \transd{a} \verb$ : $ \transd{b} \\
\transb{e} & \equiv & \ebool \verb$ ? $ \transb{a} \verb$ : $ \transb{b} \\
\transr{0}{e}   & \equiv & \ebool \verb$ ? $ \transr{0}{a} \verb$ : $ \transr{0}{b} \\
\ldots \\
\transr{R-1}{e} & \equiv & \ebool \verb$ ? $ \transr{R-1}{a} \verb$ : $ \transr{R-1}{b} \\
\end{eqnarray*}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{The simple expression $\ebool$ has boolean type, and} \\
\mbox{$a$ and $b$ are assignment convertible to a distributed array type, and} \\
\mbox{the rank of this type is $R$.}
\end{array}
\end{displaymath}
\caption{Translation of conditional operator selecting distributed arrays.}
\label{fig:transConditionalOperator}
\end{figure}

\clearpage

\subsection{Translating the conditional operator}

The general scheme is illustrated in Figure \ref{fig:transAssignment}.
The definition of the result type, and associated conditions
about legality of selecting between $a$ and $b$, follow from this
translation, in conjunction with the normal Java rules for types of
conditional expressions, applied term by term to the parts of
the translated arrays.


% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & a \verb$.dat()$
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transv{e} & \equiv & \transd{a}
\end{eqnarray*}

\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & a \verb$.bas()$
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transv{e} & \equiv & \transb{a}\verb$.base$
\end{eqnarray*}

\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & a \verb$.grp()$
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transv{e} & \equiv & \transb{a}\verb$.group$
\end{eqnarray*}

where:
\begin{displaymath}
\begin{array}{l}
\mbox{The simple expression $a$ has distributed array type.} \\
\end{array}
\end{displaymath}
\caption{Translation of distributed array inquiries.}
\label{fig:transArrayInquiries1}
\end{figure}

\clearpage

\begin{figure}[tbp]

\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & a \verb$.rng($r\verb$)$
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transv{e} & \equiv & \transr{r}{a}\verb$.range$ \\
\end{eqnarray*}

\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & a \verb$.str($r\verb$)$
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transv{e} & \equiv & \transr{r}{a}\verb$.stride$ \\
\end{eqnarray*}

where:
\begin{displaymath}
\begin{array}{l}
\mbox{The simple expression $a$ has distributed array type, and} \\
\mbox{the term $r$ is a compile-time constant integer expression} \\
\quad \quad \mbox{in the range $0 \le r < R$, where $R$ is the rank of $a$.}
\end{array}
\end{displaymath}
\caption{Translation of distributed array inquiries, continued.}
\label{fig:transArrayInquiries2}
\end{figure}


% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & \verb$($ T \verb$ [[$ \mbox{\it attr}_0 \verb$, $ \ldots \verb$, $ \mbox{\it attr}_{R-1} \verb$]] $ \mbox{\it bras} \verb$) $ a
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transd{e} & \equiv & \verb$($ T \verb$ [] $ \mbox{\it bras} \verb$) $ \transd{a} \\
\transb{e} & \equiv & \transb{a} \\
\transr{0}{e}   & \equiv & {\it CAST\_DIMENSION}\left(\mbox{\it attr}_0,
                                     \transr{0}{a} \right) \\
\ldots \\
\transr{R-1}{e} & \equiv & {\it CAST\_DIMENSION}\left(\mbox{\it attr}_{R-1},
                                     \transr{{R-1}}{a} \right) \\
\end{eqnarray*}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{$T$ is a Java type,} \\
\mbox{each term $\mbox{\it attr}_r$ is a single hyphen, {\tt -}, or a single asterisk, {\tt *},} \\
\mbox{the term $\mbox{\it bras}$ is a string of zero or more bracket pairs, {\tt []},} \\
\mbox{$a$ is an array expression in the source program, of rank $R$, and} \\
\mbox{the macro {\it CAST\_DIMENSION} is defined in the text.}
\end{array}
\end{displaymath}
\caption{Translation of cast of array expression.}
\label{fig:transArrayCast}
\end{figure}

\clearpage

\subsection{Translating casts}
\label{sec:transCasts}

The scheme for translating a cast of a distributed-array valued
expression is illustrated in Figure \ref{fig:transArrayCast}.
The macro {\it CAST\_DIMENSION} is defined as follows:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it CAST\_DIMENSION}\left(\mbox{\it attr}_r, a'_r\right) \equiv$
    \verb$ (ArrayDim) $$a'_r$
\end{tabbing}
\end{bigCode}
\end{displaymath}
if the term $\mbox{\it attr}_r$ is empty, or
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
$\mbox{\it CAST\_DIMENSION}\left(\mbox{\it attr}_r, a'_r\right) \equiv$
\verb$ (SeqArrayDim) $$a'_r$
\end{tabbing}
\end{bigCode}
\end{displaymath}
if the term $\mbox{\it attr}_r$ is an asterisk, {\tt *}.

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & a \verb$ instanceof $ T \verb$ [[$ \mbox{\it attr}_0 \verb$, $ \ldots \verb$, $ \mbox{\it attr}_{R-1} \verb$]] $\mbox{\it bras}
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transv{e} & \equiv & \verb$($ \transd{a} \verb$ instanceof $
                             T \verb$ [] $ \mbox{\it bras} \verb$) && $ \\
           &        & \verb$($ \transr{0}{a} \verb$ instanceof $ 
                      {\it DIMENSION\_TYPE}\left(\mbox{\it attr}_0\right)
                                            \verb$) && $ \\
           &        & \ldots \\
           &        & \verb$($ \transr{R-1}{a} \verb$ instanceof $ 
                      {\it DIMENSION\_TYPE}\left(\mbox{\it attr}_{R-1}\right)
                                            \verb$)$
\end{eqnarray*}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{$a$ is an array expression in the source program, rank $R$,} \\
\mbox{$T$ is a Java type,} \\
\mbox{each term $\mbox{\it attr}_r$ is a single hyphen, {\tt -}, or a single asterisk, {\tt *},} \\
\mbox{the term $\mbox{\it bras}$ is a string of zero or more bracket pairs, {\tt []},} \\
\mbox{the macro {\it DIMENSION\_TYPE} is defined in the text.}
\end{array}
\end{displaymath}
\caption{Translation {\tt instanceof} applied to array expression.}
\label{fig:transArrayInstanceof}
\end{figure}

\clearpage

\subsection{Translating {\tt instanceof}}

The scheme for translating an {\tt instanceof} test applied to 
a distributed-array valued expression is illustrated in Figure
\ref{fig:transArrayInstanceof}.
The macro {\it DIMENSION\_TYPE} is in section
\ref{sec:transArrayDeclaration}.

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{eqnarray*}
e & \equiv & a \verb$ == $ b
\end{eqnarray*}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transv{e} & \equiv & \verb$($ \transd{a} \verb$ == $
                               \transd{b} \verb$) && $ \\
           &        & \verb$($ \transb{a} \verb$ == $
                               \transb{b} \verb$) && $ \\
           &        & \verb$($ \transr{0}{a} \verb$ == $ 
                               \transr{0}{b} \verb$) && $ \\
           &        & \ldots \\
           &        & \verb$($ \transr{R-1}{a} \verb$ == $ 
                               \transr{R-1}{b} \verb$)$
\end{eqnarray*}

\textbf{SOURCE:}
\begin{displaymath}
a \verb$ != $ b
\end{displaymath}

\textbf{TRANSLATION:}
\begin{eqnarray*}
\transv{e} & \equiv & \verb$($ \transd{a} \verb$ != $
                               \transd{b} \verb$) || $ \\
           &        & \verb$($ \transb{a} \verb$ != $
                               \transb{b} \verb$) || $ \\
           &        & \verb$($ \transr{0}{a} \verb$ != $ 
                               \transr{0}{b} \verb$) || $ \\
           &        & \ldots \\
           &        & \verb$($ \transr{R-1}{a} \verb$ != $ 
                               \transr{R-1}{b} \verb$)$
\end{eqnarray*}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{Expressions $a$ and $b$ are assignment compatible with a distributed array type, and} \\
\mbox{this type has rank, $R$.} \\
\end{array}
\end{displaymath}
\caption{Translation of reference equality tests, applied to array expressions.}
\label{fig:transArrayEquality}
\end{figure}

\clearpage

\subsection{Translating reference equality}

The scheme for translating the reference equality operators
\verb$==$ and \verb$!=$, applied to distributed arrays is illustrated
in Figure \ref{fig:transArrayEquality}.

Because there is no single Java reference to a distributed array (or,
equivalently, because a distributed array is not considered to be an
object) there is no obvious, a priori definition for reference equality
between distributed arrays.  The translations given here can be read
as definitions.

One consequence of these definitions, together with the translation for
array sections given in section \ref{sec:transArraySections}, is that an
array section in which every subscript is the default triplet \verb$:$
is considered to be identical (in the sense of reference equality)
to its parent array.

{\em [Define the value of {\tt null} here??  Have to define it somewhere.]}


\clearpage

\section{Support Classes}

Figures \ref{fig:HPspmdInterface} through \ref{fig:DADClass}
list the most important parts of the various classes and interfaces
used in the HPJava translation scheme.

The classes defined in figures \ref{fig:HPspmdInterface} through
\ref{fig:TripletClass} can be freely used in an HPJava source program.
The classes defined in figures \ref{fig:ArrayBaseClass} through
\ref{fig:DADClass} are effectively reserved by the HPJava translator:
they will appear in generated code, but should not appear in a program
input to the translator.

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public interface HPspmd {
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt HPspmd} interface.\label{fig:HPspmdInterface}}
\end{figure*}


\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public abstract class Group implements HPspmd, Cloneable {
    public boolean member() {...}

    public boolean contains(Dimension d) {...}

    public void restrict(Dimension d) {...}
    public void restrict(Dimension d, int crd) {...}

    public Object clone() {...}
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt Group} class.\label{fig:GroupClass}}
\end{figure*}

\clearpage

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public class Procs extends Group {
    public Procs(int [] shape) {...}

    public Procs(int rank) {...}

    public Dimension dim(int r) {...}
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt Procs} class.\label{fig:ProcsClass}}
\end{figure*}

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public class Procs0 extends Procs {
    public Procs0() {
        super(0) ;
    }
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt Procs0} class.\label{fig:Procs0Class}}
\end{figure*}

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public class Procs1 extends Procs {
    public Procs1(int p) {
        super(new int [] {p}) ;
    }

    public Procs1() {
        super(1) ;
    }
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt Procs1} class.\label{fig:Procs1Class}}
\end{figure*}

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public class Procs2 extends Procs {
    public Procs2(int p1, int p2) {
        super(new int [] {p1, p2}) ;
    }

    public Procs2() {
        super(2) ;
    }
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt Procs2} class.\label{fig:Procs2Class}}
\end{figure*}

\clearpage

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public abstract class Range {
    public int size() {...}

    public int bas() {...}
    public int str() {...}

    public int volume() {...}

    public int format() {...}

    public Dimension dim() {...}

    public CollapsedRange shell() {...}

    public int loExtension() {...}
    public int hiExtension() {...}

    public Range subrng(int lo, int hi) {...}
    public Range subrng(int lo, int hi, int stp) {...}

    public Location location(int n) {...}

    public Block localBlock() {...}
    public Block localBlock(int lo, int hi) {...}
    public Block localBlock(int lo, int hi, int stp) {...}

    public Triplet crds() {...}
    public Block block(int crd) {...}

    public Triplet crds(int lo, int hi) {...}
    public Block block(int crd, int lo, int hi) {...}

    public Triplet crds(int lo, int hi, int stp) {...}
    public Block block(int crd, int lo, int hi, int stp) {...}

    public boolean containsLocation(Range x, int n) {...}

    public boolean containsLocations(Range x) {...}
    public boolean containsLocations(Range x, int lo, int hi) {...}
    public boolean containsLocations(Range x, int lo, int hi, int stp) {...}

    public ArrayDim arrayDim(int stride) {...}

    static final DIST_DIMENSION = ... ;
    static final DIST_BLOCK     = ... ;
    static final DIST_CYCLIC    = ... ;
    static final DIST_COLLAPSED = ... ;
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt Range} class.\label{fig:RangeClass}}
\end{figure*}

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public class Dimension extends Range {
    public int crd() {...}
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt Dimension} class.\label{fig:DimensionClass}}
\end{figure*}

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public class BlockRange extends Range {
    public BlockRange(int size, Dimension d) {...}
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt BlockRange} class.\label{fig:BlockRangeClass}}
\end{figure*}

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public class ExtBlockRange extends Range {
    public ExtBlockRange(int size, Dimension d, int wLo, int wHi) {...}
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt ExtBlockRange} class.\label{fig:ExtBlockRangeClass}}
\end{figure*}

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public class CollapsedRange extends Range {
    public CollapsedRange(int size) {...}
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt CollapsedRange} class.\label{fig:CollapsedRangeClass}}
\end{figure*}

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public class CyclicRange extends Range {
    public CyclicRange(int size, Dimension d) {...}
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt CyclicRange} class.\label{fig:CyclicRangeClass}}
\end{figure*}

\clearpage

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public class Block {

    public final int count ;

    public final int sub_bas ;
    public final int sub_stp ;

    public final int glb_bas ;
    public final int glb_stp ;

    public Block(int count, int sub_bas, int sub_stp,
                            int glb_bas, int glb_stp) {
        this.count   = count ;

        this.sub_bas = sub_bas ;
        this.sub_stp = sub_stp ;

        this.glb_bas = glb_bas ;
        this.glb_stp = glb_stp ;
    }

    public static final EMPTY = new Block(0, 0, 1, 0, 1) ;
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt Block} class.\label{fig:BlockClass}}
\end{figure*}

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public class Location {
    public final int crd ;
    public final int sub ;

    public Location(int crd, int sub) {
        this.crd = crd ;
        this.sub = sub ;
    }
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt Location} class.\label{fig:LocationClass}}
\end{figure*}

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public class Triplet {
    public final int lo ;
    public final int hi ;
    public final int stp ;

    public Triplet(int lo, int hi, int stp) {
        this.lo  = lo ;
        this.hi  = hi ;
        this.stp = stp ;
    }

    public boolean inRange(int n) {...}
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt Triplet} class.\label{fig:TripletClass}}
\end{figure*}

\clearpage

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public class ArrayBase {
    public final Group group ;
    public final int base ;

    public ArrayBase(Group group, int base) {
        this.group = group ;
        this.base  = base ;
    }
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt ArrayBase} class.\label{fig:ArrayBaseClass}}
\end{figure*}

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public class ArrayDim {
    public final Range range ;
    public final int stride ;

    public ArrayDim(Range x, int stride) {
        this.range  = x ;
        this.stride = stride ;
    }
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt ArrayDim} class.\label{fig ArrayDimClass}}
\end{figure*}

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public class SeqArrayDim extends ArrayDim {
    public final int off_bas ;
    public final int off_stp ;

    public SeqArrayDim(CollapsedRange x, int stride) {
        super(x, stride) ;

        off_bas = stride * (x.loExtension() + x.bas()) ;
        off_str = stride * x.str() ;
    }

    public SeqArrayDim(int size, int stride) {
        super(new CollapsedRange(size), stride) ;

        off_bas = 0 ;
        off_stp = stride ;
    }
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt SeqArrayDim} class.\label{fig:SeqArrayDimClass}}
\end{figure*}

\begin{figure*}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
public class DAD {
    ArrayBase base ;
    ArrayDim [] dimensions ;
}
\end{verbatim}
\end{code}
\end{displaymath}
\caption{The {\tt DAD} class.\label{fig:DADClass}}
\end{figure*}

\end{document}
%--------------------------------------------------------
\clearpage

\section{Optimization}

\subsection{On subscripting}

\subsection{Analysis of {\tt overall} constructs}

If an array is subscripted with a distributed index, as in:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$    at(i = $$x$\verb$ [$$n$\verb$]) {$ \\
\verb$      ... $$a \mbox{\tt [}e_0\mbox{\tt ,} \ldots
                    \mbox{\tt ,} e_{r - 1}
                    \mbox{\tt ,} i
                    \mbox{\tt ,} e_{r + 1} \mbox{\tt ,} \ldots
                    \mbox{\tt ]}$ \verb$ ...$ \\
\verb$    }$
\end{tabbing}
\end{bigCode}
\end{displaymath}
or
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$    overall(i = $$x$\verb$ for $$l$\verb$ : $$u$\verb$ : $$s$\verb$) {$ \\
\verb$      ... $$a \mbox{\tt [}e_0\mbox{\tt ,} \ldots
                    \mbox{\tt ,} e_{r - 1}
                    \mbox{\tt ,} i
                    \mbox{\tt ,} e_{r + 1} \mbox{\tt ,} \ldots
                    \mbox{\tt ]}$ \verb$ ...$ \\
\verb$    }$
\end{tabbing}
\end{bigCode}
\end{displaymath}
then, according to section \ref{sec:elementAccessRules} the expression
$a$ should be an side-effect free invariant.  In fact, according to
the policies defined section \ref{sec:subscriptChecking}
it should be simply named, final variable, declared outside the {\tt overall}
construct.

Consider the nested {\tt overall} and loop constructs from Figure
\ref{fig:directMatmul}, which we reproduce here:
\begin{displaymath}
\begin{code}
\begin{verbatim}
    overall(i = x for :)
      overall(j = y for :) {

        float sum = 0 ;
        for(int k = 0 ; k < N ; k++)
          sum += a [i, k] * b [k, j] ;

        c [i, j] = sum ;
      }
\end{verbatim}
\end{code}
\end{displaymath}
A correct but naive translation of this fragment, based on the
translation scheme in Figure \ref{fig:localOverall}, is given in
Figure \ref{fig:naiveDirectMatmul}.

\begin{figure}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
    Block bi = x.localBlock() ;
    
    for (int lx = 0 ; lx < bi.count ; lx++) {
      Block bj = y.localBlock() ;

      for (int ly = 0 ; ly < bj.count ; ly++) {

        float sum = 0 ;
        for(int k = 0 ; k < N ; k++)
          sum += a.dat() [a.bas() + 
                         (bi.sub_bas + bi.sub_stp * lx) * a.str(0) +
                          k * a.str(1)] *
                 b.dat() [b.bas() +
                          k * b.str(0) +
                         (bj.sub_bas + bj.sub_stp * ly) * b.str(1)] ;

        c.dat() [c.bas() +
                (bi.sub_bas + bi.sub_stp * lx) * c.str(0) +
                (bj.sub_bas + bj.sub_stp * ly) * c.str(1)] = sum ;
      }
    }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Naive translation of {\tt overall} from Figure \ref{fig:directMatmul}}
\label{fig:naiveDirectMatmul}
\end{figure}

Actually this translation isn't quite general because it exploits
some specific assumptions about the sequential dimensions of {\tt a}
and {\tt b}.  But in this section we will concentrate on subscripting in
{\em distributed} dimensions.  The main problem we want to address here
is the complexity of the associated terms in the subscript expressions.
Specifically we want to replace the code of Figure
\ref{fig:naiveDirectMatmul} with something like the code in 
Figure \ref{fig:optimizedDirectMatmul}, where the subscript expressions
have been greatly simplified by application of {\em strength-reduction
optimizations}.

\begin{figure}[tbp]
\begin{displaymath}
\begin{code}
\begin{verbatim}
    Block bi = x.localBlock() ;
    
    int vai_ = a.bas() + bi.sub_bas * a.str(0) ;
    int vci_ = c.bas() + bi.sub_bas * c.str(0) ;

    final int sia0 = bi.sub_stp * a.str(0) ;
    final int sic0 = bi.sub_stp * c.str(0) ;

    for (int lx = 0 ; lx < bi.count ; lx++) {
      Block bj = y.localBlock() ;

      int vb_j = b.bas() + bj.sub_bas * b.str(1) ;
      int vcij = vci_    + bj.sub_bas * c.str(1) ;

      final int sjb1 = bj.sub_stp * b.str(1) ;
      final int sjc1 = bj.sub_stp * c.str(1) ;

      for (int ly = 0 ; ly < bj.count ; ly++) {

        float sum = 0 ;
        for(int k = 0 ; k < N ; k++)
          sum += a.dat() [vai_ + k * a.str(1)] *
                 b.dat() [vb_j + k * b.str(0)] ;

        c.dat() [vcij] = sum ;

        vb_j += sjb1 ;
        vcij += sjc1 ;
      }

      vai_ += sia0
      vci_ += sic0
    }
\end{verbatim}
\end{code}
\end{displaymath}
\caption{Translation of {\tt overall} from Figure
\ref{fig:directMatmul}, after applying strength reduction to
distributed index subscript expressions}
\label{fig:optimizedDirectMatmul}
\end{figure}

The specific goal in this version was to eliminate complicated expressions
involving multiplication from expressions in inner loops.  This was
achieved by introducing the {\em induction variables}:
\begin{eqnarray*}
\verb$vai_ $ & \equiv &
    \verb$a.bas() + (bi.sub_bas + bi.sub_stp * lx) * a.str(0)$ \\
\verb$vci_ $ & \equiv &
    \verb$c.bas() + (bi.sub_bas + bi.sub_stp * lx) * c.str(0)$ \\
\verb$vb_j $ & \equiv &
    \verb$b.bas() + (bj.sub_bas + bj.sub_stp * ly) * b.str(1)$ \\
\verb$vcij $ & \equiv &
    \verb$c.bas() +$
\begin{array}[t]{ll}
    \verb$(bi.sub_bas + bi.sub_stp * ly) * c.str(0) +$ \\
    \verb$(bj.sub_bas + bj.sub_stp * ly) * c.str(1)$
\end{array}
\end{eqnarray*}
which can be computed efficiently by incrementing at suitable points with the
{\em induction increments}:
\begin{eqnarray*}
\verb$sia0 $ & \equiv & \verb$ bi.sub_stp * a.str(0)$ \\
\verb$sic0 $ & \equiv & \verb$ bi.sub_stp * c.str(0)$ \\
\verb$sjb1 $ & \equiv & \verb$ bj.sub_stp * b.str(1)$ \\
\verb$sjc1 $ & \equiv & \verb$ bj.sub_stp * c.str(1)$
\end{eqnarray*}
In Figure \ref{fig:optimizedDirectMatmul} we deliberately
refrained from reducing the multiplications {\tt k * a.str(1)} and
{\tt k * b.str(0)} associated with subscripting sequential dimensions.
In general such an optimization depends on analysis of the sequential
{\tt for} loop involved; here we are specifically interested in
{\tt overall} constructs and distributed index subscripts.

In rest of this section we will to describe a compile-time algorithm
for identifying the induction variables and induction increments need
to optimize an arbitrary program in this manner.

We define an {\em indexing pattern} as an mapping
of the members of a set of distributed indices to a subset of the
dimensions of a particular array.  A {\em dimension assignment}%
\footnote{{\em Not} in the sense of an assignment statement or expression!}
an association of an individual distributed index with an individual
dimension.

A dimension assignment of a particular index can be represented by a
tuple $(i, a, r)$ where $i$ is an index, $a$ is an array name and $r$ is a
dimension label in the range $0, \ldots, R-1$, where $R$ is the rank of $a$.
An indexing pattern can be conveniently represented by a list of dimension
assignments.

Now we assume the compile-time description of an index $i$ is annotated
with a {\em partial indexing set} and a {\em dimension assignment set}.
As the names suggest, the former is a set of indexing patterns, and the
latter is a set of dimension assignments.

These sets are initialized to the empty set for every index in the
program, then the program text is traversed to look for subscripting
expressions---distribu\-ted array accesses or array section expressions.

When a subscripting expression is encountered the following procedure
is executed.  We assume the array name is $a$, and $p$ is a variable
in which the indexing pattern for this expression will be accumulated:
\begin{enumerate}
\item
Set $p$ to {\tt null}.
\item
Find the distributed index with {\em largest scope} appearing
in any subscript of the expression.  
\item
\label{item:indexingStart}
Suppose this index is $i$ and it appears as a subscript in dimension $r$.
Look for the tuple $(i, a, r)$ in the dimension assignment set
of $i$.  If it doesn't appear, add it to the set.
\item
Set $p$ to $(i, a, r) \verb$++$ p$, ie, a list with head
$(i, a, r)$ and tail $p$.
\item
Look for the pattern $p$ in partial indexing
pattern set of $i$.  If it doesn't appear, add it to the set.
\item
\label{item:indexingEnd}
Find the distributed index with {\em next} largest scope appearing
in any subscript.  Repeat steps \ref{item:indexingStart} to
\ref{item:indexingEnd} until all indexes appearing in the
subscripting expression have been exhausted.
\item
Annotate the compile-time description of the subscripting expression with $p$.
\end{enumerate}

As a specific example, in the matrix multiplication code from the start
of this section, the partial indexing patterns associated with
{\tt i} will be:
\begin{eqnarray*}
& [(\mbox{\tt i}, \mbox{\tt a}, 0)] \\
& [(\mbox{\tt i}, \mbox{\tt c}, 0)]
\end{eqnarray*}
and the partial indexing patterns associated with {\tt j} will be:
\begin{eqnarray*}
& [(\mbox{\tt j}, \mbox{\tt b}, 1)] \\
& [(\mbox{\tt i}, \mbox{\tt c}, 0), (\mbox{\tt j}, \mbox{\tt c}, 1)]
\end{eqnarray*}
These might equivalently be written as:
\begin{eqnarray*}
& \mbox{\tt a} [\mbox{\tt i}, -] \\
& \mbox{\tt c} [\mbox{\tt i}, -]
\end{eqnarray*}
and
\begin{eqnarray*}
& \mbox{\tt b} [-, \mbox{\tt j}] \\
& \mbox{\tt c} [\mbox{\tt i}, \mbox{\tt j}]
\end{eqnarray*}
respectively.

After the whole program has been processed in this way, the translation
of the {\tt overall} construct associated with a particular index involves
defining one induction variable for every member of the partial indexing
set of that index, and one induction increment for every member of the
dimension assigment set.  The details are given in the next section.

% Want figures to start on even pages, so descriptive text starts opposite
% (Unfortunately there is no even-page equivalent of \cleardoublepage.)
\clearpage
%\verb$ $
%\newpage

\begin{figure}[tbp]
\textbf{SOURCE:}
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$overall ($$i$\verb$ = $$\erng$\verb$ for $
                       $\elo$\verb$ : $$\ehi$\verb$ : $$\estp$\verb$) $$S$ \\
\end{tabbing}
\end{bigCode}
\end{displaymath}

\textbf{TRANSLATION:}

\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
\verb$Block $$b$\verb$ = $
          $\erng$\verb$.localBlock($$\transv{\elo}$\verb$, $
                                   $\transv{\ehi}$\verb$, $
                                   $\transv{\estp}$\verb$) ;$ \\
\\
\verb$Group $$p$\verb$ = (Group) $$\mbox{\it apg}$\verb$.clone() ;$ \\
$p$\verb$.restrict($$\erng$\verb$.dim()) ;$ \\
\\
${\it DEFINE\_INDUCTION\_VARIABLES}\left(i, b\right)$ \\
${\it DEFINE\_INDUCTION\_INCREMENTS}\left(i, b\right)$ \\
\\
\verb$for (int $$l$\verb$ = 0 ; $
$l$\verb$ < $$b$\verb$.count ; $
                                 $l$\verb$++) {$ \\
\\
\verb$    $$\trans{S}{p}$ \\
\\
\verb$    $${\it INCREMENT\_INDUCTION\_VARIABLES}\left(i\right)$ \\
\verb$}$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where:
\begin{displaymath}
\begin{array}{l}
\mbox{$i$ is an index name in the source program,} \\
\mbox{$\erng$, $\elo$, $\ehi$, and $\estp$ are expressions in the source,} \\
\mbox{$S$ is a statement in the source program,} \\
\mbox{$b$, $p$ and $l$ are names of new variables, and} \\
\mbox{the macros {\it DEFINE\_INDUCTION\_VARIABLES},} \\
\quad \quad \mbox{{\it DEFINE\_INDUCTION\_INCREMENTS}, and } \\
\quad \quad \mbox{{\it INCREMENT\_INDUCTION\_VARIABLES} are defined in the text.}
\end{array}
\end{displaymath}

\caption{OptimizedTranslation of {\tt overall} construct.}
\label{fig:optTransOverall}
\end{figure}

\clearpage

\subsection{Translating {\tt overall} constructs}

A translation for the {\tt overall} construct is given in Figure
\ref{fig:optTransOverall}.  This assumes the existence of several
``translation functions''.  The function on
expressions, ${\bf T}_{\mbox{\small val}}\left[e\right]$, returns the
result of translating an expression $e$.  We may expect that in general
such translation involves the execution of a series of statements
as well as the evaluation of a final expression\footnote{These
statements may, for example, assign values to temporary variables}.
The sequence of statements that needs to be executed before ${\bf
T}_{\mbox{\small val}}\left[e\right]$ is evaluated is given by the
function ${\bf T}_{\mbox{\small pre}}\left[e\right]$.

The variants ${\bf T}_{\mbox{\small val}}^V$, ${\bf T}_{\mbox{\small
pre}}^V$ defined a translation of $e$ that always returns a simple name
for ${\bf T}_{\mbox{\small val}}^V\left[e\right]$, by having
${\bf T}_{\mbox{\small pre}}^V\left[e\right]$
define an extra temporary variable for the result if necessary.

The macro {\it DEFINE\_INDUCTION\_VARIABLES} is defined as follows:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
${\it DEFINE\_INDUCTION\_VARIABLES}\left(i, b\right) \equiv$ \\
\verb$            $$v_i$\verb$ = $$b$\verb$.glb_bas ;$ \\
\verb$            $${\it DEFINE\_INDUCTION\_VARIABLE}\left(p_0, b\right)$ \\
\verb$            ...$ \\
\verb$            $${\it DEFINE\_INDUCTION\_VARIABLE}\left(p_{P-1}, b\right)$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $v_i$ is the name of a new temporary variable,
$\{p_0, \ldots, p_{P-1}\}$ is the partial indexing set
attached to $i$, and
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
${\it DEFINE\_INDUCTION\_VARIABLE}\left(p, b\right) \equiv$ \\
\verb$                $$v_p$\verb$ = $$\mbox{\it base}$\verb$ + $$b$\verb$.sub_bas * $$a$\verb$.str($$r$\verb$) ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}
Here $v_p$ is the name of a new temporary, and the
index/array/dimension-label tuple $(i, a, r)$ is the dimension assignment
in the {\em head} of the partial indexing pattern, $p$.  If the tail,
$p'$, of $p$ is empty, then
\begin{displaymath}
\mbox{\it base} = a\verb$.bas()$
\end{displaymath}
Otherwise, if $p'$ is non-empty,
\begin{displaymath}
\mbox{\it base} = v_{p'}
\end{displaymath}
Here $v_{p'}$ is the name of an induction variable associated with $p'$.
By construction of the partial indexing patterns, $v_{p'}$ will have
been defined earlier in the translated program.

The macro {\it DEFINE\_INDUCTION\_INCREMENTS} is defined by:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
${\it DEFINE\_INDUCTION\_INCREMENT}\left(i, b\right) \equiv$ \\
\verb$            $$s_i$\verb$ = $$b$\verb$.glb_stp ;$ \\
\verb$            $${\it DEFINE\_INDUCTION\_INCREMENT}\left(d_0, b\right)$ \\
\verb$            ...$ \\
\verb$            $${\it DEFINE\_INDUCTION\_INCREMENT}\left(d_{D-1}, b\right)$
\end{tabbing}
\end{bigCode}
\end{displaymath}
where $s_i$ is the name of a new temporary variable,
$\{d_0, \ldots, d_{D-1}\}$ is the set of dimension assignments
attached to $i$, and
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
${\it DEFINE\_INDUCTION\_INCREMENT}\left(d, b\right) \equiv$ \\
\verb$                        $$s_d$\verb$ = $$b$\verb$.sub_stp * $$a$\verb$.str($$r$\verb$) ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}
Here $s_d$ is the name of a new temporary variable, and $(i, a, r)$ is the
index/array/dimension-label tuple in the dimension assignment, $d$.

The macro {\it INCREMENT\_INDUCTION\_VARIABLES} is defined as follows:
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
${\it INCREMENT\_INDUCTION\_VARIABLES}\left(i\right) \equiv$ \\
\verb$            $$v_i$\verb$ += $$s_i$\verb$ ;$ \\
\verb$            $${\it INCREMENT\_INDUCTION\_VARIABLE}\left(p_0\right)$ \\
\verb$            ...$ \\
\verb$            $${\it INCREMENT\_INDUCTION\_VARIABLE}\left(p_{P-1}\right)$
\end{tabbing}
\end{bigCode}
\end{displaymath}
Here $v_i$, $s_i$ are the names introduced above for the induction
variable and increment of the global index value,
as above $\{p_0, \ldots, p_{P-1}\}$ is the set of partial indexing patterns
attached to $i$, and
\begin{displaymath}
\begin{bigCode}
\begin{tabbing}
${\it INCREMENT\_INDUCTION\_VARIABLE}\left(p\right) \equiv \quad v_p$\verb$ += $$s_d$\verb$ ;$
\end{tabbing}
\end{bigCode}
\end{displaymath}
Here $v_p$ is the name of the induction variable associated with $p$, and
$s_d$ is the name of the temporary holding the increment for the dimension
assignment, $d$, contained in the {\em head} of $p$.

Now the translation of a subscripting expression that is annotated with
indexing pattern $p$ will involve the value of the induction variable
$v_p$.  The details of this translation are properly speaking part of
the translation functions for expressions, which will be described later.

The {\em shift step} for the index $i$ is defined to be the value
of $\erng$\verb$.str()$.  This value is used in computation of offsets
associated with shifted index subscripts.

%\bibliography{HPJava}

\end{document}

