\setcounter{equation}{0}
\setcounter{figure}{0}

\chapter{A General Software Integration Framework}

As we discussed in the previous chapter, a general semantic model of program
execution is needed that can serve as a framework and programming
paradigm for the
integration of program components written in high-level parallel languages or 
in portable message-passing
systems on a meta-computer. In this chapter we first illustrate 
how in dataflow concepts functional programming
ideas may be directed to exploit the power of parallel and distributed systems
and have the potential to lead to an important universal semantic model
of computation and high-level, coarse-grained software execution model. 
We then propose
a general software integration framework based on the dataflow model. We describe in such a
framework the major features needed, the advantages a user can gain, and
the key issues and requirements to be addressed in building the framework.
At the end of the chapter we review similar work in a parallel software integration
framework and give
a detailed description of the Application Visualization System, a framework
testbed to be used in later chapters to develop various application
systems and to apply concepts in the proposed integration model to
certain more specific software integration contexts.

\section{Introduction to Dataflow Programming}

Studies in dataflow programming have been active ever since the 
limitations of the conventional von Neuman architecture were found
in the 1970's \cite{bacus,dataflow1}. As an alternative to and departure from 
the control-driven programming model inherent in the von Neuman 
architecture, dataflow models have been extensively 
studied and practiced in various fundamental areas of computer science, including
computation models, programming languages, machine architecture, compilers, and parallelism. Many
early researchers in this area believed that the dataflow model
would become a new and general model of computation, be able to
exploit all aspects of parallelism arising in general purpose programs,
and that it would enable the development of high-level languages in which
the programmer need not manage every detail of efficient mapping
of programs and data on a parallel machine \cite{dataflow2}. Most
research and papers have been devoted to the architecture and language
aspects of a dataflow model in fine-grained parallelism and program execution.
Indeed, the dataflow concept has been found in many applications
in conventional computing such as the pipelining and 
multiple instruction issue techniques used in many
RISC processors. Dataflow programming was once a promising approach
to a new generation of high-performance computing. 
The reality is that it is still not mature
enough to be a main-stream technology in general parallel computing,
and it is not widely accepted and practiced in the HPDC community and industry.
It is anticipated that
von Neuman-based processors, on which current parallel technology is built,
will continue to dominate high-performance computing and applications
for many years to come.

However, the dataflow concept has many advantages and unique characteristics
that can be borrowed to use in today's parallel software
environment. We believe that a hybrid parallel software system,
combined with a conventional control-driven model and dataflow programming,
will likely complement each other and be able to 
provide better software and hardware solutions to
parallel software for general application problems that require high-performance
computing. One of the major findings in this thesis
shows that a dataflow model can fit naturally into an HPDC software engineering
environment as a unified software
integration framework.  In this chapter we focus our discussion
on adopting a dataflow model as a programming paradigm and execution
model.

\subsection{Dataflow Model As a General Programming Paradigm}
Writing parallel programs is currently much more difficult than
writing sequential programs. The difficulty comes mostly from three aspects:

\begin{enumerate}
\item
Managing the control flow along multiple threads of execution
\item
Managing the placement and access of global data
\item 
Synchronizing events in such a way that program execution is deterministic
and parallel speedup is achieved
\end{enumerate}

While it is possible for a programmer or end-user to 
choreograph explicitly a parallel program by indicating which tasks execute
when, how tasks communicate and synchronize, and where
data should be placed and how it can be accessed, such an undertaking is
extremely complex.  

Dataflow and data-driven are radical departures from the traditional
approach based on the von Neumann or a control-driven computation model.
In the von Neumann model, a piece of processor sequentially fetches
instructions and data from memory for execution. In the dataflow execution model, a
program is represented as a graph in which nodes can be viewed as virtual processors.
They execute as soon as the input operands become available on the input edges.
There is no concept of memory. The concept
of memory in a control-driven model introduces states in which every update
of a memory location implies a state transition. This state-driven
nature of the von Neumann model makes it a sequential model. In the
data-driven model, a piece of data is not represented as a static
entity (memory) whose value can be updated time after time.
Instead, it is a dynamic entity that is produced once and consumed
by another instruction (fine-grained dataflow) or by a set of instructions,
i.e., by a program entity (coarse-grained dataflow). This is illustrated
in Figure \ref{model1}.

\begin{figure}
\centerline{\epsfxsize=450pt \epsfbox[50 500 550 720]{chapter2-general-model/model1.eps}}
%BoundingBox: 50 530 550 720
\label{model1}
\caption{The von Neumann model vs. a dataflow model}
\end{figure}

In the data-driven model shown in Figure \ref{model1}, 
a program is represented as a directed acyclic
graph in which nodes represent instructions and edges represent
the data dependency relationship between the connected nodes. A data
element produced by a node flows on the outgoing edge(s)
to the connected node(s) for consumption. Node executability is determined
by the firing rule, which says that a node can be scheduled for
execution if and only if its input data become valid for
consumption. In this computing abstraction, every node is
purely functional in that it causes no side-effects. Also, computation is
inherently parallel and asynchronous, since any number of nodes can be 
executed as long as their input data become valid.


\begin{figure}
\centerline{\epsfxsize=450pt \epsfbox[50 530 550 720]{chapter2-general-model/model2.eps}}
%BoundingBox: 50 530 550 720
\caption{A coarse-grained dataflow model}
\label{model2}
\end{figure}

A dataflow model can be applied to a fine-grained instruction level in which
the basic execution unit is a machine instruction. A fine-grained dataflow
requires the support of a programming language and machine architecture
for problem representation, efficient program mapping, implementation and
execution. On the other hand, a dataflow model can also be used as a unique and
simple programming concept at the coarse-grained level in which the basic
execution unit is a program module; the message-passing among
distributed program modules is restricted in a data-driven semantics. 
In the coarse-grained dataflow execution model, a distributed application
consists of a graph of modules. Each module may run on a different or the same
physical processor and the message passing between any two modules is through
the edges. The execution of any module in the graph depends solely on the
availability of  the input operands.
While different
dataflow systems require different support in hardware and software
environments, we are most interested in adopting dataflow as
a general coarse-grained programming paradigm and methodology for
the purpose of software and system integration on HPDC systems.

The major advantage of a dataflow model as a general parallel programming
paradigm lies in the following features:

\begin{enumerate}
\item
Data dependencies among programs are represented as data channels
that connect programs. Message passing between programs is only via
data channels. Each program consists of 
two distinct and relatively separated components:
computation and communication. They can be encapsulated and
object-oriented for both computation and communication parts.
A dataflow-based parallel program is close to a sequential
program. Simplicity is the key feature in dataflow programming.
\item 
The execution of a program is activated only by the availability
of the data object it depends on. Thus the overall computation is
inherently parallel and asynchronous.
\item
Mapping of a program to a processor is determined solely at runtime
by system software.
It is also only at runtime that a channel between programs is established
to represent the logical relationships among the programs of an application. Thus
the overall programming can be much closer to the problem representation
and frees the programmer from knowing the underlying
machine architecture and implementation details, while it is the system software's
job to choreograph and optimize tasks that are closer to the machine
implementation, such as program-to-processor mapping, task scheduling, and
coordination.

\item
Because there is almost no dependency of one program on another, except
for the data objects and data types that flow in and out of a program,
software modularity and reusability is highly preserved in the system.
\end{enumerate}

Explicit message passing is the major way in which most current
parallel programming systems produce efficient interprogram communications,
synchronization, task scheduling, and representation of
data dependency among parallel programs.
We can view dataflow programming as a restricted message-passing system.
In a dataflow model, message passing between two programs is via the
flow channel connecting the output ports of an upstream program (A)
to the input ports of a downstream program (B). Those ports are predefined
so that only data of the same type can be passed.
To program A, the message passing of one instance of a data object
out of the program is asynchronized so that the computation part
of program A can be resumed soon after it puts data on its output ports.
To program B, the communication can be viewed as blocked, as it must
wait on its input ports for incoming data before it can start a new
computation. The dataflow model that requires message passing flows
only through the channels based on the firing rule.

\subsection{Dataflow As the Overall Program Execution Model}

The message-passing-based integration model is currently the most effective means
for the execution of distributed modules on a parallel system,
especially if the modules run on a heterogeneous distributed computing system and
are written in different languages. Apart from the synchronization
requirement in distributed program execution, functional requirements of
a distributed application also make it inevitable that messages will be transfered
among different functional components that usually are physically
located in different places. For example, it is a common practice
to use one machine to collect runtime input data, another
to process computation, and yet a third one to display results. 
At the application system specification level, work-flow together with 
dataflow has been widely used in
system analysis to derive functional system definition and to 
structurally model the conceptual representation of problem
components and their relationships \cite{dataflow4}.
As a restricted message-passing model, the dataflow model can be
used to define the synchronization semantics of a message-passing system
so that the modules' execution is data-driven with communications of two modules  
strictly through the dataflow channels with no other means 
allowed to pass data between modules.

As shown in Figure \ref{model2}, when a coarse-grained dataflow model is applied to a 
distributed application where most of the program modules are located on different
physical processors loosely interconnected by a local network,
it lends itself to a naturally suitable program execution model with a restricted
message-passing mechanism for general-purpose distributed and parallel
applications. This is due to the following features found in a 
typical HPDC application:

\begin{enumerate}

\item
{\bf Task-parallel programming in coarse-grained parallelism at topmost level}.
When we are dealing with a distributed computing system with the scale
of a meta-computer where all computing units are loosely connected
by one or more local networks with moderate bandwidth, the unit ratio
of computation to communication performance at the highest level
of the system is expected to be high. Message passing is through 
the local network, instead of the
interprocess communication or shared-memory implementation when all modules
are located on the same machine. Thus the tasks, like
the automous processing nodes, have to be coarse-grained and relatively independent
in terms of the frequency of message passing among each other.
There will a clear programming and execution hierarchy of granularity,
abstraction/expressiveness, and efficiency in the system.

\item
{\bf Functional programming requirements in task decomposition of an application}.
Most HPDC applications require a computing resource that may have to
be physically located over different hardware/software components, and usually
they are event-driven, such as window manager, I/O, or GUI. A dataflow execution
best matches the event-driven models in a distributed environment in order
to optimize
overall resource utilization and performance.

\item
{\bf Modules for an HPDC application likely written in different languages or
programming models}. Using a dataflow model to facilitate the interfaces among
modules provides a uniform, simple, and portable high-level interface 
to ``hook up'' diverse program
interfaces that support interoperability among various programming tools. 

\end{enumerate}

\section{A General Integration Framework Based on a Dataflow Model}

Based on our analysis and observation of the generic requirements
posed by a meta-problem and the system and software characteristics
of a meta-computer in Chapter 1, together with the nice 
features found in a dataflow execution model,
we propose a simple and effective dataflow-based software
integration model as the overall framework for meta-computing.
We outline the common underlying abstractions and concept
for ``a module interconnection framework'' that uses a dataflow execution
mechanism to interconnect distributed
coarse-grained parallel software components.

Intuitively, a software integration framework can be viewed as
a software bus that presents a standard interface
into which distributed modules may be ``plugged.'' In the same way that
a hardware bus presents a standard for electrical characteristics
and signal protocols --- so that boards with that standard may 
be plugged together --- a software integration framework interconnects
software components whose internal properties may remain private as long as its
interface matches the ``bus'' standard.

More specifically, a software integration model is a communication facility
among separate distributed modules for a HPDC application. The abstract
model is a specification of the interface provided by this
 facility, and the model implementation shows how those specifications
are to be realized for a particular set of programming languages
and models, host platforms, and communication media. In this way,
decisions concerning the interfacing of modules may be
encapsulated separately from the modules, rather than distributing
those decisions among the application modules themselves.

We describe an abstracted integration model in this section, leaving the model
implementation to the rest of the thesis.
This integration framework will be further demonstrated under a different system and application
context and through extensive case studies and prototypes in the later
chapters.

The design goal of an integration architecture includes:
\begin{enumerate}
\item
{\bf Ease of use}. It should support such tools as a module plug-in, template, and a visual programming interface or 
network editor to compose the flow network visually by ``drag-plug.''
\item
{\bf Software modularity}. Software reuse is one of the major features required
in a software support environment. Another important feature is 
system extensibility, which should provide for fully automatic parallelization of \\
trivially/embarrassingly parallel algorithms (with no dependencies).
\item
{\bf Portability and scalability}.
This will provide portable API and a paradigm for developing distributed
application applications in this integration model on a meta-computer
to enable users to make efficient use of distributed resources.
\end{enumerate}

The integration model has the following key components:

\begin{enumerate}
\item
{\bf An execution architecture}.
An execution model must be defined to specify how distributed
modules communicate with one another through synchronization.
We propose a coarse-grained dataflow model in which each
module is represented as a node in the flow graph, which is
in turn an abstraction of the 
flow execution and process control
of an application. 
Dataflow architecture is composed of
modules, ports, channels (connections of ports) as well as
an execution manager that is the kernel program
to ensure proper execution in a dataflow paradigm.
Data is communicated between modules through ports. Each
output port has a specific data type, and
each input port has a collection of data types it can
accept. When connecting 
an output port of one module to an input of another 
(usually via a visual programming
interface), the user interface queries the target module to
discover whether it accepts the type produced. If not, an error
is signaled. Output from a single port can be 
sent to multiple destinations. Each destination
is notified when a change occurs and
can request data as needed. In some cases,
multiple output ports can also be connected to a single
port. All of the modules connected to this port are treated
identically. Data is cached, or buffered, for all modules on the
output ports to reduce
unnecessary recomputation, and it can be stored with 
some reference count to avoid unnecessarily copying.

\begin{figure}
\centerline{\epsfxsize=450pt \epsfbox[50 380 550 720]{chapter2-general-model/flow.eps}}
%BoundingBox: 50 530 550 720
\caption{Sequential dataflow vs. concurrent dataflow}
\label{flow}
\end{figure}


Because the targeted parallel hardware infrastracture
is a meta-computer, 
the number of
dataflow piplelines in this software framework,
unlike in a single machine environment,
 is more than one and
the overall flow execution is in a concurrent fashion so that
task parallelism is achieved at the topmost framework level (Figure \ref{flow}).


The dataflow between modules in this framework is
demand-driven in that data computation occurs only
as modules downstream request recomputation of the data.
These recomputations typically are triggered by external
events such as new source data arriving or interactive parameter
changes that require
a new output to be generated. The change is first reported
to the module whose parameter is affected and  
subsequently reported downstream until it reaches a terminal
module. Because of this notification, the terminal
module indicates to the system that its outputs are out of date and
at some time in the future it is called to update itself.
The recomputation can also be requested 
by the readiness of an output device, in which case
the terminal module generates new requests for data as 
often as the device is ready to accept new data.

Because the overall framework is on a distributed meta-computer,
to support parallelism and distributed functionality
a module that needs new inputs requests them from all its
upstream connections and then waits for them to finish
their computation. The effect is a ripple
of requests back through the network from the 
terminators and then a ripple of computation forward toward the
terminators. When all of the data objects for a module 
are ready, the user-supplied computation procedure is called.
The execution of upstream modules and collection of
the resulting data (on the input ports) is invisible
to the module writer. All the module sees
is that its computation procedure is called with up-to-date input.
The degree of concurrency in the framework
depends on the fan-out and fan-in to each
module in the network. If a                           
module has three inputs, each of them
can run concurrently. If each of those three
modules in turn has two inputs, all six
of those modules can run concurrently.


The flow executive (manager) is the central component of the
system that determines when to run various modules.
Its implementation can be like a
distributed operating system
scheduler. It basically needs to maintain two resources: 
a queue of modules that need execution, and a pool of tasks,
or threads of control, that can be assigned to execute on
one of the modules.

The implementation of our architecture is largely modeled
by the AVS data-driven model \cite{avs2}, to be discussed further in this chapter.
Indeed, we will use AVS as the test bed for most of our case studies
conducted in this work. We believe, however, that the framework proposed here
has a more general meaning than the original AVS model when it is applied to
a broader meta-computer context and HPDC application domain.

\item
{\bf Modules}. 
Modules are the software components of the distributed
system and the building blocks that can be interconnected  
to build a parallel application. They are coarse grained and 
at a higher level of abstraction
than the fine-grained procedures/functions, but at a lower level than the
complete application. The abstraction of applications, which consists
of smaller components that can be customized and reused, has become
common practice in software engineering and is a theme of
object-oriented programming technology. The notion of modules is
extremely important to a distributed environment where they encapsulate
functional components that are independent of platforms and locations.
Program behavior is the focal point of the integration framework
in such an abstract way that modules are ``black-boxes'' that consume
input data to produce output data. They 
can be interconnected to form computation networks for particular tasks
where the distributed modules can execute in parallel.
In summary, modules receive typed data as inputs and produce typed
data as outputs. The only way the other modules can observe or affect
a module is through input or output ports/data defined for the module.
The rest of the details of the module are hidden. Thus this framework
presents to an application builder an abstract interface
via module input/output ports.

In addition to input and output ports, modules can also have parameters
used to control the module's computation. The parameters attached to
a control panel can be accessed via a GUI or any other
input peripherals. The parameters used primarily for modules
require an 
interactive graphical input source other than that
from the normal dataflow input ports.

Modules can be characterized by their input and output connections
as follows:

\begin {enumerate}
\item
{\it Source Modules}. They have no inputs and one or more outputs. They
represent such data sources as database access programs or simulations that
are producing data directly within the system. Another example is
a simulator program that continously generates random data.

A source module can be a distributed program to take run-time input
from a GUI, a file, an RDBMS, or any other source that feeds a data stream into
the application. In most cases, specially customized programming efforts
are 
required to gather input from different sources. The behavior of source
modules is mostly event- or data-driven in nature and, except where parallel
I/O is used, they are serialized by the source supplier.

\item
{\it Computation Modules}. They accept one or multiple inputs from source or
other computation modules and perform certain computations that 
generate one or more outputs. A computation module can be any
module that is relatively independent of other modules in the application.
Inter-module communication occurs only at the beginning and end of the
module computation.

\item 
{\it Terminal Modules}. They have one or more input connections
and no outputs. They represent the outputs of the application,
such as graphic display, file, or database write output.
\end{enumerate}

A number of tools should be provided to construct the modules'
``glue'' automatically, including a visual programming interface that
can be used by developers to configure application flow networks
from module executables that are precompiled and stocked. 
In particular, each module
can be described, starting with a template file and a name for
the module, its input and data types, and the parameters that can
be accessed from the control panel. This approach requires module
writers to implement only the functionality required by their
algorithm in the module. Inputs and outputs
can be represented by arguments based on the description in the
template. The module writer or integrator will usually
write/integrate the computation procedure/subroutine that
performs the fine-grained parallel computations unique to the module.

\begin{figure}
\centerline{\epsfxsize=450pt \epsfbox[50 420 550 720]{chapter2-general-model/module.eps}}
%BoundingBox: 50 80 550 720
\label{module}
\caption{Internal structure of a module}
\end{figure}

Figure \ref{module} gives an internal conceptual structure of a module.


\item
{\bf Application programming interface (API)}.
Based on the
integration architecture, an API is needed to allow
developers to specify the interface (communication) part of an module
that can be plugged into the framework. Wrappers and
templates for different languages are the common means used
to migrate existing programs to the framework.
API is usually provided in the forms of libraries, micros and
pre-processing tools. Because the dataflow model is a very
restricted message-passing system, its simplicity makes
it possible to develop automatic code generators that can be
used by developers to wrap existing modules and to generate
the interface code segment for new modules.

\item
{\bf A message-passing protocol in dataflow}.
The protocol defines when and how data objects are
transferred reliably between two dataflow modules that can be either
on either the same or different machines. It should guarantee
a synchronized data transfer and program execution in a
deterministic dataflow computation semantics.

While it is implemented as part of the execution runtime system
using TCP/IP as the base data transport protocol, there
can be many approaches in implementing this message-passing
protocol. In general, message passing on the same host should
use the most efficient mechanism possible, such as
shared memory or message queues. Reliability and performance
are the major considerations in the actual implmentation
over different configurations of distributed systems.
The base protocol should be message-based, rather than stream-based,
so that the most recent input messages received for a module
can be buffered by the runtime system.

\item
{\bf Data objects}.
Data entities flow over pipelines connecting
different modules on distributed systems. In the dataflow
framework, data objects are strongly typed and usually predefined.
The system should define a set of basic data types, and a mechanism should be
provided to allow user-defined data types by composing
new data types from the basic and existing ones.
The data objects will also be used/consumed within a module.
Because message passing is restricted to dataflow channels,
the data objects, types, and operations can be well defined and restricted to enforce
strict type-checking and operation constraints.

In graphics applications like AVS,
such data are merely predefined data structures or
data types by C micros or a set of API
libraries. They are designed for particular
filter or transform operations of data visualization/graphics,
as most system and user-developed modules deal with graphics-related
data such as field, geometry, or number.

In a relational database management system such as Oracle7,
tables/relations are the basic data objects in the dataflow execution
model. Operations on relational tables are thus well-defined
relational operators such as select, join, project, and union.

In the above two examples, the data object is only a data structure
or non-object-oriented data, as not all the operations (methods) defined
to operate on those objects are attached to (encapsulated in) 
the data structure definition itself, but are provided in the form 
of separate constructs in the programming environment, 
no matter how atomic they may be.

Different applications may require different types of data objects. For
a general class of applications, a truly object-oriented data structure may be
the best candidate in this framework to ensure that the dataflow programming is really
an ``object-flow'' integration framework. For example, 
Java \cite{java} can be well used as an API to define object-oriented data that are
most suitable to WWW applications.

\item
{\bf A dataflow network runtime support facility}.
A visual programming interface contains the kernel of
the runtime facility. It is the GUI for creating, revising, and
starting a dataflow network that represents a distributed
application.

One of the major advantages in coarse-grained
dataflow programming is the abstraction of ``port,'' ``channel,''
and ``module.'' It provides programming constructs that allow a
stand-alone program to be wrapped up and encapsulated, so that
its data dependency can be visually represented and configured
through a ``drag-plug'' graphical interface.

A constructed flow network via this interface should be captured
within a text or script language. This language is used to drive
the deployment and runtime execution of the modules on a particular
meta-machine configuration. The mapping from a flow network
onto a physical distributed system should be done at runtime
so that certain scheduling optimization and fault-tolerant execution
can be enforced.

\item
{\bf A module publication/registration mechanism}.
A module repository should be established 
to allow module developers to publish, contribute, and
share standard and user-defined dataflow modules. Module
reuse is an important means for improving the productivity of 
distributed software development. The dataflow module provides a uniform
standard interface to facilitate portable software authoring.

\end{enumerate}

Traditional data-parallel and message-passing interfaces have been
shown to give a high performance, but the lack of existing third
party software for MPP machines and workstation clusters
demonstrates the insufficiency of these interfaces in ease of use. 

Several distinct styles of parallelisms can be seamlessly
integrated and are important to address in the framework:

\begin{enumerate}
\item
{\bf Data parallel}. In the data-parallel programming style, the data set is decomposed into 
chunks that are distributed among several nodes. In this framework
each node processes its subset of the data essentially independently, perhaps
passing messages among nodes to synchronize computations or exchange data.
Note that this represents something of a departure from 
traditional usage in that a data-parallel model in the framework
may still use message passing, and
it does not require a data-parallel compiler as HPF does.

This framework is ideally suited to this programming model, since
the data sets are usually in a known, regular, self-describing format.
This simplifies the task of decomposing the data set, managing the
computations, and locating off-node parts of the data.

\item
{\bf Pipeline parallel}. There can be two types of pipeline parallel
application in the framework: (1) time-step granular where, for example,
a time step of a simulation is processed by a single module and then sent
to the next module, potentially freeing the first module to accept
the next time step, and (2) chunk granular where the data set is read in or otherwise produced in
pieces (either fractions of time-steps or unrelated to time-steps
boundaries), and those pieces sent down the pipe. The chunk-granular method
is more useful when the data set sizes for each time step are large compared
to the memory of the machine(s). Of course, there are obvious restrictions
in the type of processing that can take place in a chunk-granular pipeline
parallel application; the processing cannot depend on data yet to be seen,
for instance. However, it is always possible to use a demand-driven
or pull (as opposed to push) architecture to overcome some of these
limitations. In this configuration the last module demands a certain portion
of the data set from the previous module(s) until some module is able
to satisfy it from data it has locally, or which it is able to compute. We can
use the dataflow dependence graph to implement this kind of parallelism by
using the same kind of data set decomposition as in the data-parallel style.

\item
{\bf Explicit message passing}. The message-passing approach is simpler, more standard, and will generally
give a better performance by giving more
control to the user. In this framework this can largely be subsumed by
the expanded data parallel model. If the user desires, all data transfer
can be done via message passing, eliminating the automatic data
decomposition. This may be appropriate for some applications, but
in most cases allowing the system to distribute the data will save effort
and time.

\item
{\bf Shared memory}. The fourth type of parallelism is the shared-memory model. In 
this model, data is logically stored in one address space with one or
more task runs on each node sharing the data. Data may be
physically shared, as in an SMP machine, or virtually shared, in which
case page faults on a given node result in a page transfer from 
another node.

Several supports for shared memory can be provided under this framework
such as spawn parallel tasks/process instances, manage synchronization,
and support message-passing library; the user is free to
use whatever shared-memory system is supported on specific 
hardware. This framework makes no attempt to unify this model with
the message-passing model beyond presenting a uniform task and
data decomposition model, which will limit the portability of modules
written explicitly for a shared-memory model only to shared-memory machines.

Since message passing on a shared-memory machine simply involves pointer copying,
which is relatively fast, a message-passing model is recommended even
for shared-memory machines.

\end{enumerate}

\section{Similar Work in Software Integration on HPDC Systems}

In this section, in order to illustrate alternatives that can be
used in building a software integration framework, we compare the approaches of three primary systems in the
context of parallel programming and software engineering environments and
integration capability. They are:

\begin{enumerate}
\item
AVS
\item
PVM
\item
Fortran-M
\end{enumerate}

AVS \cite{avs1} is a widely available commercial visualization environment based on 
a dataflow model for scientific data visualization and process control. It
incorporates visualization, graphics, visual programming, process management,
and transparent networking into a single comprehensive visualization
software and development environment. The AVS dataflow is a model of
parallel computation in which a flow network of autonomous processes compute
by passing data along arcs that interconnect them by input/output
ports of the process modules. Each process module fires autonomously
as soon as all the input data it requires arrive on its output ports,
thus triggering further computation. Both AVS process control and data transfer
are in a modular fashion and are completely transparent to
the programmer. AVS provides a data-channel abstraction that transparently
handles module connectivity and ports type-checking.
The module programmer needs only to define input and output ports
in predefined AVS data types by using a set of AVS library
routines and micros. Message passing occurs at a high level of
data abstraction and only through the input and output ports. An AVS
kernel (manager) executes the flow network and coordinates the actual
data transfer, which are actually carried out by an underlying IPC or RPC mechanism.
When the flow network is executed on a uniprocessor machine,
IPC (Inter-Process Communication) protocol is employed. When it is
executed under a distributed environment, such as a cluster of workstations
in which each process or group of processes may execute on a different
machine, RPC (Remote Procedure Call) and TCP/IP sockets are used for
message passing in the network. Among other systems similar to AVS are
Explorer from SGI \cite{iris}, Data Explorer from IBM \cite{dataexplorer}, and
Khoros \cite{khoros}.

PVM \cite{pvm} is a message-passing programming system for the development and
execution of large concurrent or parallel applications that consist
of many interacting but relatively independent components. It can
operate on a collection of heterogeneous computing elements (nodes)
interconnected by one or more networks. PVM provides a straightforward
and general interface that permits the description of various types
of algorithms and their interactions, while the underlying infrastructure
permits the execution of an application on a virtual computing environment
that supports multiple parallel computation models. PVM contains facilities
for a concurrent, sequential, and conditional execution of application components.
Programs in PVM are portable to a variety of architectures. PVM provides
a set of message-passing primitives that can be incorporated into
existing procedural languages. Primitives exist for invoking processes, 
message transmission and reception, broadcast, synchronizations, etc.

Other systems include MPI \cite{mpi} and Express.

Fortran-M \cite{fortran-m} is a small set of extensions to Fortran77 that supports a modular
approach to the execution of sequential and parallel
programs. It allows the specification of dynamic processes and communication
structures for the integration of task and data parallelisms in an application.
In a programming language context, Fortran-M provides constructs
for creating tasks and message-passing channels, for sending and receiving
messages among processes, and for mapping tasks and data to processors.
Program processes communicate via established channels.
Channels and processes can be created dynamically during the execution.
Support of concurrent program modularity that is enforced at the language level and
task parallelism are the major motivation and features in Fortran-M.

Similar systems are CC++ \cite{cc++} and CSP \cite{csp}.

Besides specific capability in their own strengths, namely, AVS for data
visualization, PVM for portable message-passing program development, and
Fortran-M for developing modular task parallel programs, all three
systems can be used to integrate different services and parallel software
components into a meta-problem. However, the capability and approach each
system has adopted for this purpose differ
significantly in the aspects of system configuration, integration
architecture, abstraction of message passing, and the level of integration
abstraction.

\begin{enumerate}

\item 
{\bf Processor and Task Configuration}.
Fortan-M has language constructs (semantics) with which a program
defines when and how a virtual processor array and a process (i.e., task)
array are mapped to each other at runtime. The user
program must explicitly declare the mapping.
A compiler runtime takes care of the allocation
of physical processors.
The parallel program model
in Fortran-M is MPMD (Multiple Program Multiple Data), while in PVM
the default model is SPMD (Single Program Multiple Data), in which
the allocation and mapping of processes and processors are 
implicitly carried out by PVM daemons. A user's program does not need to
handle them explicitly. Instead of built-in language constructs and
semantics, as in Fortran-M, PVM provides a set of process management
library routines to allow new tasks and processors to be
dynamically added/removed from the runtime virtual machine
configuration. The module and machine allocation and binding in AVS
are configured statically through AVS's click-and-point visual
programming interface called Network Editor. It must be done
before the execution of the flow network, as the network configuration
is not allowed to change during the execution, although a configuration of the
flow network can be saved in a text file for future reuse and modification.

\item 
{\bf Integration Architecture}.
AVS uses a centralized and process-based model in which modules in separate
memory address space are loosely coupled by a central kernel manager. This
manager controls and coordinates all the data transfer and overall program
execution. PVM uses a distributed and loosely coupled process-based
execution model in which tasks on each machine have a unique daemon manager
that supervises message passing and interactions with other daemon
managers. Fortran-M ties the integration model into the language runtime.
It is a tightly coupled non-process-based system.

\item 
{\bf Abstraction of Message-passing}.
AVS and Fortran-M use a highly abstracted ``port'' and ``channel'' concept to
represent/define how communication is established and data transfered in the
program. A channel is a connection between an input port of a module
and an output port of another module such that the only way to communicate
among modules is via established channels. A port is a predefined
data object associated with a module
with certain data types. It is strongly typed so that
only data with the same type as the port can be attached.
There are only two types of ports.
Input ports are used to receive input values from outside of the module, while
output ports generate new values. 
Only ports of the same type can be connected to form a channel.
To establish a channel is to connect an input port of an module to an
output port of another module. Since there is no specific identification
of source or destination modules in a program, realization of a networking
configuration can be delayed until the system execution, to be dynamically
determined according to resource availability. PVM has no such abstraction
of module, data, and connection. Every send/receive operation must explicitly
specify the source/destination task ID, although the ID is also a logical
representation of a node in a virtual machine.

\item 
{\bf Level of Integration}.
AVS is an integrated software development framework under which
coarse-grained modules of different modules on a distributed system
can be equally combined in the pipeline to solve a complex problem. 
Message passing in the framework is restricted in a dataflow
fashion and must be coarse-grained. Modularity and many other
conventional software engineering features such as reusability,
object-oriented design, extensibility, software sharing, 
productivity, and rapid prototyping can easily be
achieved in the AVS environment. It can be a generic integration
framework to be extensible to parallel and distributed
environment. 

Fortran-M attempts to be an extended parallel programming language
to support the integration of task and data parallel modules at a
language level. It can be either coarse-grained or fine-grained, but
the most natural and efficient programming in Fortran-M is a coarse-grained task program to embed fine-grained data-parallel modules.

PVM is a message-passing library that can be used to develop
integration tools. It is the most flexible of them
and provides fine granularity, yet it is basically
a set of primitive routines and an execution runtime for
explicit message-passing programming. No modularity is assured and almost 
everything related to higher-level software engineering facilities
has to be built from scratch.

\end{enumerate}

\section{Application Visualization System --- A Testbed for
Software Integration in HPDC Applications}

AVS \cite{avs1} is a software development toolkit
based on a dataflow model
for scientific data visualization and process control.
It incorporates visualization, graphics, 
visual programming, process management, and networking
into a single comprehensive software  
development environment. We are most interested in its software
integration capability. 

AVS dataflow is a model of parallel computation in which a flow
network of autonomous processes computes by passing object data along arcs 
interconnected by input/output ports. Each module, represented
as an individual UNIX process, fires autonomously 
as soon as any requires new input object arrives
on its input ports. It then produces an output object that flows on
its output port, thus triggering further computations. A set of base
communication objects is predefined, such as numbers and fields
($n$-dimensional arrays), and a collection of computational modules is
provided that transforms sets of input objects into sets of output objects.
AVS itself offers an extensive library of ready-to-use modules primarily meant to be used
for graphics programming and filtering, as well as for specifying the
protocol for developing and including user modules into the system. 
Specific applications are assembled from reusable modules with the
aid of visual network editing tools. Modules are visualized as
nodes, ports as terminals, and communication objects as links of an
interactively constructed dataflow network. 

Both process control and data transfer in AVS, are in a 
modular fashion and are completely transparent to the programmer. 
AVS provides a data-channel abstraction that 
transparently handles module connectivity and ports type-checking.
The module programmer needs only to define the input and output ports
in AVS-predefined data types and use a set of AVS routines and micros. 
Message passing occurs at a high level of data abstraction and 
only through the input and output ports. 
Each AVS application includes an AVS kernel and a set of modules
specified by a dataflow network. An AVS kernel executes network
specification and instantiates module processes as well as
lower-level intermodular communication links based on
the RPC/XDR UNIX network protocol. Another important feature of the
AVS execution model is that it supports a remote module execution
in which various nodes of the computational graph can be executed on
distributed machines on a local or wide area network.

