%\setcounter{chapter}{2}
\setcounter{equation}{0}
\setcounter{figure}{0}

\chapter{Integrating Multiple Parallel Programming Paradigms}

\markright{CHAPTER 3. MULTIPARADIGM PARALLEL PROGRAMMING}

An integrated software environment that supports different
programming paradigms and languages is important for solving a complex
HPDC problem on a meta-computer. In this chapter
we investigate how parallel programs written in different and 
multiple parallel programming models can be combined under
the proposed single integration framework to solve an application.

By viewing different parallel programming paradigms as an essentially 
heterogeneous approach to mapping ``real-world'' problems to parallel
systems, we discuss methodologies for integrating multiple programming
models on massively parallel and distributed systems.
Using a dataflow-based integration 
model built in the AVS testbed, we describe a simple,
effective, and modular way to combine sequential, data-parallel, and
explicit message-passing modules into an integrated parallel programming
environment. Three case studies are chosen from numerical
advection modeling, computational electromagnetics, and financial modeling,
respectively, to illustrate our approach in multi-paradigm programming
integration.

\section{Introduction}

A major issue in parallel processing is
homogeneous versus heterogeneous at both software and hardware levels.
There are currently
two mainstream approaches in parallel software development to deal with this issue. 
One attempts to hide the heterogeneity of architectures from
the programmer. This approach usually employs 
software technology, especially advanced 
compiler techniques, to present users with a homogeneous, high-level
programming model. This methodology leads to highly portable programs 
across multi-platforms of different architectures and programming models
and it greatly simplifies programming on parallel machines.
However, the compiler design is extremely complex, and it is
difficult to 
obtain efficient programs for general
application on a particular machine.
Examples of this approach include the FortranD project
by Northeast Parallel Architectures Center
at Syracuse University and Rice University \cite{fox2}, various automatic tools,
and the efforts of HPF \cite{hpf} and the emerging MPI message-passing standard \cite{mpi}.
The other approach acknowledges heterogeneity at both software and
hardware levels by extending existing programming languages and 
software environments into parallel environments, thus exploiting
the high performance of underlying architectures. 
The advantage of this approach is the resultant highly efficient programs 
and an open software environment flexible enough to support
the integrated heterogeneous processing environment 
proposed in Chapter 2.

While the first approach is promising and represents a major 
direction in parallel programming language development, we believe 
that the future of high-performance computing lies in the integration of 
existing and emerging parallel
architectures and technologies into a heterogeneous networked computing
environment, able to solve general classes of applications with
various requirements in scientific computation, communication, 
programming models, hierarchy memory, I/O, data processing and visualization. 
Another advantage of this approach is its ability to adopt existing programs in which
only computationally intensive components need to be rewritten to run on 
supercomputers. This is especially important in porting a large existing 
application system when a significant proportion of the sequential program are 
``interface or management codes,'' i.e., program segments that do not perform real 
scientific computational tasks but simply manage various system resources
such as standard I/O, graphical user interface, system control 
interface, OS or file system interface, and networking interface, to name
a few. This heterogeneous approach, together with our view of parallel 
software as a mapping process between
two complex systems --- problems and parallel computers \cite{fox1} --- would
require integration methodologies and tools to facilitate multiple parallel 
programming models in an unified environment. Software integration is 
expected to play a critical role in parallel software development and 
applications.

\section{Multiparadigm Parallel Programming Environments}

For this thesis we shall consider that a {\em programming paradigm} 
is a model or 
an approach employed in solving a programming problem with a restricted
set of concepts. Programming paradigms either induce or are induced
by a class of languages, the programming environments in which these languages
are supported, or the software engineering discipline one uses to produce
systems within these environments. Typically, each class places different
requirements on its associated programming environment \cite{hail}.

\subsection{Multi-paradigm Programming in Mapping a Problem 
Domain to a Parallel System}

On existing parallel systems, there are two dominant 
parallel programming paradigms:
\begin{enumerate}

\item
{\bf Data parallel}, in which more or less the same operation is 
performed on many data elements by many processors simultaneously.
The data parallel program exploits parallelism
in proportion to the quantity of data involved, hence it offers the
highest potential for concurrency. It is also
easy to write, understand, debug, and port.

\item
{\bf Explicit message passing}, which allows two or more operations to be
performed simultaneously. A message-passing program 
is free to arbitrarily distribute 
program tasks and data, using 
synchronization and data communication between nodes when necessary. 
The message-passing paradigm allows the programmer maximum flexibility 
and control in exploiting domain decomposition and in controlling 
decomposition of the application problem so that maximum 
efficiency utilizing an MPP hardware can 
be achieved through careful design of explicit message-passing 
parallel algorithms.  
 
\end{enumerate}

Like the trade-offs among various conventional
programming languages, expressiveness and efficiency are the two major 
considerations in different parallel programming models. The trade-off is even more 
significant, because the driving force using
a parallel computer is the performance requirement, and programming on parallel 
machines tends to be more difficult and less portable. 
Programming languages for massively 
parallel processors must go
beyond conventional languages in expressivity and the cost/performance ratio. Unfortunately, 
unlike sequential machines and due to efficiency 
considerations on a particular SIMD/MIMD architecture with a
unique interconnection topology, 
most parallel systems support only a single programming model, i.e., 
either data-parallel or message passing. 
Given a targeted machine and an application problem, usually an early 
(long before the implementation stage) decision must
be made as to  which parallelism should be used to solve the problem.
This is
not natural and should be avoided, as (especially for large application
problems) some components may be well suited to the data parallel
paradigm, while others are suited
to the message-passing paradigm. We should exploit the parallelisms at each level of
computation. There has long been a difference of opinion in the
parallel-computing community on the issue of data-parallel
vs. message-passing programming, and SIMD vs. MIMD machines.

As studied in \cite{fox1}, by viewing ``real-world'' problems and parallel computers 
as complex systems, the fundamental issue in solving a problem 
on a parallel computer becomes whether we can easily and efficiently map 
a problem structure to a 
particular parallel architecture. A general mapping followed by a computer
simulation typically consists of several
mapping stages (Figure \ref{mapping1}). We are interested in the software
environment and programming models used to effectively map a large 
application onto a massively parallel system.

\begin{figure}
\centerline{\epsfxsize=450pt \epsfbox[50 530 550 720]{chapter3-multiparadigm/fig1.eps}}
%BoundingBox: 50 530 550 720
\caption{Computation and simulation as a series of mappings}
\label{mapping1}
\end{figure}

\begin{figure}
\centerline{\epsfxsize=470pt \epsfbox[50 500 550 720]{chapter3-multiparadigm/fig2.eps}}
%BoundingBox: 50 500 550 720
\caption{Integration of multiple programming models as a single mapping}
\label{mapping2}
\end{figure}

Parallel systems are built to solve large-scale, computationally 
intensive applications (e.g., ``grand-challenge'' problems). 
The broad range of HPCC applications may embody a rich diversity of 
program structures. 
It is anticipated that no
one parallel programming model will satisfy all needs. Data-parallel,
task-parallel, and object-parallel models have all found applicability to 
end user problems \cite{report,kim}. A useful parallel programming environment
for MPPs must incorporate semantic constructs for delineating program 
parallelism, identifying locality and specifying data set partitioning, as well
as fulfil the normal requirements of conventional software engineering.
It must permit the user access to a mix of parallelism models, 
either by incorporating them into a single schema or by providing the means for 
a single application to use different languages or different parallelism 
models for separate parts of a problem. 
In either case, a parallel software system
will of necessity support interoperability among independently
derived program modules, perhaps even running on separate computers in
a distributed heterogeneous computing environment. 
This is especially important for irregular and dynamic problems and for 
multidisciplinary applications, as it is often inevitable that those 
applications require coordination or 
composition of multiple paradigms programming (e.g., high-level data-parallel and 
low-level explicit message passing),
considering the complex of problem structures to be expressed and solved by the
software environment.  Such an example of a multi-model earth system
in which an ocean model and an atmosphere model are loosely coupled
is given in~\cite{foster}.

While the homogeneous approach 
emphasizes a virtual programming
model due largely to portability consideration, 
and high-level and low-level software are used
in a strict successive manner in the mapping shown in Figure \ref{mapping1}, 
it is most likely that a large application problem would require
programming models of different abstraction levels to be used in a hybrid
fashion. For instance, Figure \ref{mapping2} gives another
picture of how subtasks of an application task are mapped to a 
parallel system in a single mapping stage by multiple programming models.

A problem task may consist of subtasks requiring a set of 
computing services such as parallel computation/communication,
parallel database management, parallel or 
sequential I/O, real-time visualization, 
sequential event-driven graphical user interface, networking interface, etc.
This mapping process will require not only a ``universal architecture'' of the 
parallel computer to be able to efficiently support multiple programming models, but
also a software environment capable of supporting 
task-level computation/communication and of
integrating different programming 
paradigms into a single framework. Software integration techniques are clearly
required when interfacing data and control flows 
among components (modules) of different 
programming models in this single mapping process.

The CM5 is one of today's few parallel computers that support both data-parallel 
and explicit message-passing programming models. We choose it in the next sections
to evaluate the methodologies
of multiparadigm integration in this chapter. However, we believe that
many of the issues discussed are also applicable to programming 
integration in a heterogeneous computing environment in which multiple 
parallel systems are connected by high-speed LAN and/or WAN networks. 

\subsection{Multiparadigm Programming}

In this section we first examine integration 
techniques for multiparadigm programming
on a sequential machine and then   
corresponding implementations under a parallel context such as on 
a Connection Machine CM5.

On a conventional uniprocessor machine, multiparadigm integration 
can in general be divided
into the following four classes, based on the interface of how one 
paradigm is combined with the others:

\begin{enumerate}
\item
{\bf Full integration}.
A single language has constructs supporting 
multiple programming paradigms. 
The transfer of program control and data among 
different language components at runtime is implicitly arranged by the 
compiler.
The semantics of interfacing different paradigms
is built into the language specification. Usually, a programming language evolves
along this line, with a single paradigm at the initial stage and new
language components added as applications grow and different paradigms are required.
C++, while evolved from C (which
in turn evolved from Pascal) 
now supports imperative, 
modular, and object-oriented programming paradigms and can be thought of
as an 
example of this category. Another example can be found in \cite{cheng3} 
in which functional 
and logic programming paradigms are integrated into a single language.
Among those using this approach in a parallel programming language, 
CC++ \cite{cc++} and FortranM \cite{fortran-m} 
are examples that have attempted to integrate into a single language framework 
different paradigms of 
parallel programming such as data-parallel, task-parallel and 
object-parallel paradigms, 
imperative and declarative programming, shared memory, and 
message-based programs.

This approach provides the most flexible and efficient hybrid integration of 
multiple models. However, new languages have to be developed,
and a complex semantics for interfacing multiple programming paradigms
is usually inevitable. 
On an MIMD parallel system like the CM5, because data-parallel and message-passing
paradigms have dramatic implementation and performance differences 
in data mapping/processor allocation, 
task decomposition, and execution synchronization, and require
unique semantics inherently in both the programming language and underlying 
system architecture, this integration 
approach may incur interface overhead and unexpected complex semantics.
It therefore does not seem
feasible on a distributed-memory machine.

\item
{\bf Embedded integration in master-slave style}. 
The interface is a built-in construct (or convention)
in a master language to allow seamless
invocation of subroutines (functions) written in a slave language. 
No global data can be
shared directly by both languages, even though they have the same addressing 
memory space. Data
transfer among modules in master and slave languages is identical 
to that in a single
language and is carried out by passing pointers to the same shared memory
so that there is no data copy cost. 
Separate compilations of different languages are required and their object
programs are finally linearly
linked together. The executable program runs as a single process. 
This approach provides a simple, coherent, yet efficient way to support
integration at the language level. Among examples of this approach are 
inter-language calls on most conventional platforms, such as C 
calling Fortran77 or assembly, or
vice versa. This is the common approach on most SIMD and MIMD machines in order to
support the extension of parallel programming in host sequential languages,
such as the CM5 calling CMfortran, C*, or CMMD message-passing
primitives from Fortran77 or C. 


\item
{\bf Loosely coupled and process-based integration}.
Different individual program modules written in the same or different languages
are compiled and linked separately, while their executables run concurrently as
multiple
processes time-sharing a CPU. 
Message passing among program processes is
normally via socket-based interprocess communication (IPC), although if two 
modules share a host it is common to use shared
memory to optimize the communication. 
Runtime process control is coordinated by a supervising 
manager (or daemon) or the operating system. Currently,
most windowing systems and networking applications
(for instance, the client-server model) follow this approach.

On the host machine of a parallel system, IPC-based systems 
provide an open environment for
parallel programming software to be naturally migrated into the
conventional software environment in which software such as the concurrent
operating system, networking, graphics, I/O, and
sequential programming languages have been well established for many years. 
This process-based approach also opens great 
opportunities to construct a ``meta-computer''
via a networked environment \cite{smarr}.
Depending on the method (protocol) used to facilitate the IPC (such as data exchange, 
management of buffers and sockets, threading, process control and synchronization),
an integration system of this kind can be quite different. 
For example, the PVM \cite{pvm}, a process-based general message-passing system,
provides a set of user interface primitives that may be incorporated into 
existing procedural languages. 
In our work we use AVS \cite{avs2}, which is our testbed of the proposed
process-based integration framework.
We will demonstrate in later sections
the advantages of the AVS's dataflow IPC model in
facilitating data-parallel and message-passing programming paradigms on the CM5.

\item
{\bf Not-at-all integration in non-interactive style}.
Data passing and sharing among different program modules, usually 
stand-alone executables, are carried out at the (UNIX) operating system 
level through shell scripts, pipes, 
intermediate data files, and other pre- or post-processing techniques. 
This is the usual way for an application to use collectively the (parallel) 
computing services when it does not require their programming tasks to 
run in an interactive way. this
approach incurrs a high system overhead.
\end{enumerate}


\section {A Dataflow-Based Integration Approach for 
Multiparadigm Programming}

We choose two system configurations for applying the proposed integration
framework to multiparadigm programming support. One is on an MIMD
parallel system Connection Machine CM-5, and the other is on a heterogeneous
computing system where an IBM SP2 and a DEC Alpha workstation cluster
are interconnected via a LAN. In the first configuration
we describe the techniques
used to create a general programming environment that couples 
data-parallel and explicit message-passing programs in a unified
framework. The second one deals with system integration issues
to allow fine-grained PVM programs to be
seamlessly embedded in coarse-grained AVS visualization
modules in a loosely distributed environment. Several key functional
components, including parallel and distributed computing, 
interactive remote user-interface,
advanced data visualization, and transparent networking are 
combined to support real-time simulations over a heterogeneous 
computing and networking environment.

\subsection{A General Parallel Programming Environment on a CM5 with AVS}

\begin{figure}
%\centerline{\epsfxsize=370pt \epsfysize=250pt \epsfbox[0 420 612 750]{fig4.eps}}
\centerline{\epsfxsize=420pt \epsfbox[0 420 612 750]{chapter3-multiparadigm/fig4.eps}}
%BoundingBox: 0 420 612 750
\caption{A general AVS/CM5 integration model for multiparadigm programming}
\label{avs-cm5}
\end{figure}

Appendix B.1 describes the parallel programming available on the CM5 and
briefly examines the various integration approaches currently used
in the CM5 system.

Although AVS does not provide support for parallel computing within a
module, a specific module may use machine/language specific features to 
achieve parallelisms such as data parallel or explicit message passing. 
Using the modular process management in AVS and the time-sharing CMOST
on the control processor, together with CM5's parallel programming software, 
we can naturally
combine programs in the data-parallel, message-passing, and sequential
programming languages available on the CM5 into a single process-based
software system. As shown in Figure \ref{avs-cm5},
an AVS/CM5 integration framework has two distinct levels of
programming interfaces. At the higher level, an AVS dataflow model is used
to express and implement data and control flows among various modules
that represent decomposed subtasks of a specific application. 
Intermodular communication is handled by AVS IPC protocol. Process
control (e.g., synchronization and flow control), scheduling, and
realization of interactive visualization at module level are also
handled at this interface and are supervised by the AVS kernel. All the
sequential process threads at this level are executed on the same control processor
of the CM5. The AVS kernel can be on the same CP or on a different machine networked
to the CP. At the lower level, a specific CM5's parallel programming model
(for example, a CMFortran subroutine or a CMMD program) is used to implement
a highly parallelized algorithm in fine-grained parallelism, best suited to 
solve the subtask represented. All the parallel process threads at this level are
executed on the CM5's processor nodes, and the CM5's specialized data network
and control network (i.e., the fat-tree topology) are used for communications
among the processor nodes within a module. Data transfer and 
thread control between the two levels
in a process is handled by the host-node protocol in CMOST.
An AVS/CM process can be a module or a 
set of modules written in the same language/paradigm. 
A typical integrated
parallel module
will have two distinct threads at the two different levels 
to be executed on different physical processors of the CM5 system
(i.e., CP or PNs). They are represented as two different portions 
in the program code of a module:

\begin{enumerate}
\item
{\bf AVS/CM interface part}.
This is a sequential code in Fortran77, C or C++ to coordinate
data flow between AVS modules and to 
control flow between the AVS kernel and the CM parallel system. 
It contains code declaring the AVS module's
input/output ports and (graphical) parameters. For a data-parallel
module, it has program segments for packing input data, received either
from other AVS modules or by sequential subroutines in the same module
into parallel data, e.g., to transfer serial arrays into data parallel arrays
or decompose/load block data onto each processing node, or vice versa.
For an explicit message-passing module, it is a host program for the 
CMMD host/node programming model.

\item
{\bf CM computation parts}.
These data-parallel subroutines in CMFortran or C* --- or
CMMD node programs in Fortran77, C, or C++ --- to perform main calculations
requiring intensive computation and communication. This part of the module
is almost identical to a stand-alone parallel program implementing core
parallel algorithms for a subtask (if not integrated under the AVS
framework), except that it becomes a parallel subroutine invokable from
the AVS/CM interface part. 
\end{enumerate}

After both parts of a module are compiled separately, possibly with different
compilers, they are linked into a single executable. Multiple modules can either 
be compiled and linked individually into separate
executables (processes) or they can be linked into a single executable. 
Modules in different processes must communicate data through the CP under
the supervision of the AVS kernel. If the modules are written in different 
paradigms (e.g., CMFortran or CMMD), they are in different processes. 
If both modules are in the same process on the CM5, 
only a pointer needs to be passed; therefore, modules written in the same 
paradigm can be grouped in a single process and data transfer among them 
has no data-movement cost.

This integration framework provides a sufficiently high-level way to 
couple modules written in 
multiple programming paradigms into a 
single environment on a CM5, so that programmers have the 
freedom to choose either parallel programming method to solve
their problem, independently or in a mixed way. This is a 
general parallel software engineering environment that allows
more than one programming paradigm to be used, including sequential and
parallel models, data-parallel and explicit message-passing 
programming, concurrent and pipeline execution, and task
and functional parallelism. Running in an interactive environment, 
it gives the user maximum control 
over the optimization of heterogeneous resources and 
capabilities of a high-performance computing system.
Other advantages 
using this integration system include:

\begin{enumerate}
\item
{\bf Modularity and template-oriented}.
Programs are constructed by using 
explicitly declared communication channels to plug together program processes.
A process can encapsulate common data decomposition, manipulation and internal
communication, and the programming paradigm inherent in the code.
The modular approach of AVS also lends itself well to software sharing,
module reuse, extensibility, and flexibility. 

\item
{\bf Module reusability}.
CM5 data-parallel or message-passing codes and 
algorithms are modularly developed and compiled once and can then be reused 
indefinitely in a plug-and-play mode in a variety of diverse 
scientific applications. Because the interfaces between modules are 
well defined, modules developed at different computer sites
by different developers and in different paradigms may be freely shared.
Modules created for one application can readily be used in another 
similar application with little or no change.  

\item
{\bf Flexibility and extensibility for rapid prototyping}.
At the first stage of an application development, programmers can concentrate on
task decomposition, parallel algorithm design, and performance improvement and
totally ignore system integration issues. They may write their 
own modules in their favorite programming paradigm and language binding.
The programs are then migrated to the AVS environment by simply
attaching and linking the AVS interface code, while the main parallel
algorithms can remain untouched.

\item  
{\bf Hierarchical parallelisms}.
At the top of the task level decomposition of the problem domain, 
modules that are connected in the dataflow programming paradigm
have the potential to run concurrently 
or in a pipelined way. At this coarse-grained process level,
functional parallelism can also be achieved such that graphical interface, sequential
and parallel I/O, remote networking access, and the decomposed task computation
can be carried out concurrently by different components of the CM5 supercomputer.
The second level of the hierarchy is the 
hybrid data parallelism in data-parallel
modules and control parallelism in explicit message-passing modules.
At the bottom level, parallelism can be thought of
as inherent in the pipelined processing
hardware, as are the vector units in the CM5.

\item
{\bf Safety}.
Operations on communication channels are restricted so as to guarantee 
deterministic execution. Channels are strongly typed, allowing a compiler 
to check for correct usage.

\end{enumerate}


\subsection{Integration of PVM Programs into AVS Environment}

This section briefly describes how fine-grained parallel modules in
a portable message-passing system such as PVM can be naturally
incorporated into the coarse-grained dataflow modules in AVS. Background
information about the Parallel Virtual Machine (PVM) can be found in Appendix B.2.

By integrating PVM software into the AVS framework, the resulting system
offers an equally sophisticated networking and visualization functionality
with integrated networking and visualization programming interfaces.

\begin{figure}
\centerline{\epsfxsize=480pt \epsfbox[50 530 550 760]{chapter3-multiparadigm/pvm/fig1.eps}}
%BoundingBox: 0 50 612 720
\caption{The parallelism hierarchy in the AVS/PVM system}
\label{avs-pvm}
\end{figure}

There can be two basic approaches to incorporating PVM programs into 
an AVS framework. They are based on the ways in which an AVS kernel 
interacts with PVM daemons:

\begin{enumerate}
\item
Our first approach embeds a PVM programming model at a lower level into 
AVS dataflow communications at the top inter-module level.
As shown in Figure \ref{avs-pvm}, there is a parallelism hierarchy in the system at two distinct levels:
dataflow parallelism occurs among AVS visualization modules and between
an AVS module and a remote PVM (host) computation module, and a general message-passing paradigm 
is used by the PVM node tasks. A PVM node task is spawned from the 
AVS/PVM host module when the module is registered in the Network Editor. 
The AVS kernel communicates
with only one PVM host daemon through AVS input/output ports. All the 
AVS programming features (such as transparent networking, modular process management,
and event-driven dataflow) remain intact, as well as the PVM concurrent
programming paradigm. We use this approach in our case study, 
discussed in Appendix B.4. 

\item
In the second approach, within the data-flow execution of AVS modules 
PVM primitives are used by AVS remote modules to perform temporal 
synchronization of arbitrary patterns or even 
transfer data among subroutine or coroutine modules. The major purpose of PVM here
is to enhance remote process concurrency control in the AVS kernel.
In this approach a remote AVS/PVM module consists of an AVS/PVM host part
and a PVM node part. The module can arbitrarily communicate with other AVS/PVM modules, 
a host part or a node part, or even both. This communication can occur anytime 
before or after AVS
modules transfer data through normal dataflow input/output ports. 
Each AVS/PVM module executes as a relatively
autonomous process (or a group of loosely coupled processes) similar to an AVS coroutine.
The AVS kernel can communicate with as many PVM daemons as required.
This approach requires that the existing AVS kernel's interface to the Network Editor
be extended to allow a spawned AVS/PVM module from a module program to be able to
register in the Editor and to make port connections.

\end{enumerate}

\subsection {Limitations in the AVS Integration Model}

The process-based integration system offers many advantages for a
parallel software engineering environment in terms of multiparadigm programming, 
software reuse, and modularity. 
However, towards an efficient and fully functional multiparadigm
system, the current AVS framework suffers a few significant performance
problems and operational restrictions.

\begin{enumerate}
\item
{\bf Intermodular Communication}.
The major performance problem is 
a centralized AVS kernel versus a
decentralized parallel system. Restricted by the kernel and its IPC
data-flow protocol, any communication between two parallel modules
has to go through the sequential host-node channel and the module-module
pipeline on the control processor. Figure \ref{performance} illustrates this with
two modules ($m_1$ and $m_2$), each with two separate but 
linear threads ($h_1$,$h_2$ on the CP and
$p_1$,$p_2$ on the PNs). When $m_1$ needs to communicate to $m_2$, it has to 
follow the serialized data transfer
path $p_1$ $\Rightarrow$ $h_1$ $\Rightarrow$ $h_2$ $\Rightarrow$ $p_2$, while the most 
efficient path (or parallel
channel) is directly $p_1$ $\Rightarrow$ $p_2$ on PNs using CM5's high-bandwidth data
and control networks. In this sense, the current AVS kernel serializes parallel
channels. This may not be a performance bottleneck between a data-parallel
module and a message-passing module, as in most cases scalar data are
expected to flow between modules in different parallel paradigms. This
is not true for modules in the same paradigm (e.g., CMFortran modules)
between which, inevitably, very large scientific data sets must be
transferred. In addition, because each module is buffering its entire body of
data on the input and output ports and all the intermediate data have
to reside in the memory of the CP, the total memory requirement on the CP can
be very large. Continuous memory-disk swapping of processes by the
operating system can make this situation even worse.

\item
{\bf Dynamic Flow-Network Reconfiguration and Load Balance}.
The current AVS
system provides very limited support to dynamic modification of a
flow-network, e.g., processes and their communication requirements may
change with time --- processes can be created or destroyed, or a communication
pattern may move at runtime \cite{tdfl}. It is also difficult in
AVS to perform fine-grained dataflow paradigms such as a round-way dataflow,
i.e., an output port of a downstream module in reverse is connected
to an input port of a upstream module. 

In certain ``real world'' multitasking applications the need arises
to create or (re)initiate individual modules in a dynamic fashion.
Consider, for example, a signal-processing component of a command
and control system. The input dataset typically comes from several
sensors, is processed by a set of coupled filters, and is then analyzed
by a set of higher-reasoning modules. A particular specialized 
down-stream filter might be requested and activated in runtime by 
the upstream reasoning module. Such a transient specialized module
could not be visually pre-edited in the static AVS flow-network, since
it is invoked dynamically in a data-driven fashion. 

\begin{figure}
\centerline{\epsfxsize=500pt \epsfbox[50 520 550 720]{chapter3-multiparadigm/fig5.eps}}
%BoundingBox: 50 80 550 720
\caption{Serial AVS I/O vs. parallel node-node channels for intermodule communication}
\label{performance}
\end{figure}

A dynamic reconfiguration
of a flow-network is also required by load balance issues. 
The AVS/CM5 system assigns modules to processor nodes in a static
fashion and offers no support or visual metaphor for task mobility.
A good example is provided by an adaptive multi-grid PDE algorithm
operating on a highly nonhomogeneous manifold. Such an algorithm would 
start as a single data-parallel task and would often
spawn additional tasks in runtime. For instance, a complex local
geometry might be detected that requires a high-density local
mesh refinement. Local nonhomogeneities of this type will be
naturally load balanced by dynamically distributing such spots,
thereby spawning new concurrent data-parallel PDE tasks.

\item
{\bf Synchronization Control and Scheduling of Module Processes}

The only synchronization mechanism in the AVS model is passive causality
provided by the data-flow paradigm. High-performance interactive
applications often require near real-time operating system
support and tight synchronization of various media streams.
An extension of the dataflow model is required to handle such 
requirements. This includes both the underlying low-level message-passing 
system and the high-level, scriptable and visual network editing
tools. In the current model the only visual metaphor is that of a
graph connectivity, implemented in terms of nodes-modules and 
links-communications. Suitable objects and visual tokens need to be 
included to encompass the synchronization metaphor
and to provide graphical tools for representing priorities,
threads, and synchronization objects such as semaphores, mutexes, stacks,
FIFOs, or message queues.
\end{enumerate}

The performance of these systems on massively parallel machines will in
large part depend upon the systems' capability to support 
high bandwidth and low latency for data transfer between a host machine
and all the node processors, and the way in which a problem's tasks are decomposed.
Task granularity must be restricted to a coarse-grained domain, and
task partitioning should be carefully evaluated to consider the
balance of each module's computation
time and process communication time.
System software is needed to support high-bandwidth parallel I/O and shared-memory 
segments between processes. The CM/AVS system 
developed
at Thinking Machines is expected to greatly reduce the process-level, 
data-movement overhead \cite{gary,cmavs}. 
In the CM/AVS, CM5 shared-memory
regions are used to exchange data if they are in the same CM5 partition;
they exchange data directly over the routine network via a CM-domain socket
if they are in different partitions. They may communicate using parallel sockets,
with serial 
modules on a workstation over the fastest available network 
connecting the various components.

\section {Case Study I --- Comparison of Numerical Advection Models}

The first case study described in
Appendix B.3 is an earth science application. Using the process-based
framework, we developed a real-time simulation prototype in which
multiple numerical advection models written in different programming
models are coupled on a CM5 and their simulation results
graphically compared through a graphical user interface.

\section {Case Study II --- Option Price Modeling}

The second case study is a prototype simulation-on-demand system
in which parallel option-pricing models are locally implemented on 
two distributed systems, an Ethernet-connected IBM SP2 and 
a FDDI-based GIGswitch-connected DEC Alpha farm, and then coupled
with an interactive graphical user interface over an ATM-based
wide area network. This integrated networking/visualization framework 
allows us to use a high-level 
software approach to combine AVS visualization modules 
with remote concurrent PVM tasks in fine-grained parallelism.
Details of this case study are in Appendix B.4.

\section {Case Study III --- Distributed Computational Electromagnetics Systems}

In the last case study we describe the development and
implementation of a computational electromagnetics
application on multiple parallel and distributed systems.
This application requires the decomposition of problem tasks into
different tightly coupled program components to be written
in different parallel programming models in order to achieve an optimized
balanced performance when running on a targeted meta-computer. We also
describe a framework that embeds a resource scheduling component into
the proposed dataflow integration model.

The complete application is given in Appendix B.5.

\section {Conclusion}

The purpose of integrating multiparadigm programming
is to provide programmers with a rich and attractive programming environment 
in which a choice of high- and low-level programming models can be made for
problem structures, as well as particular machine architectures, 
to facilitate a gradual and eventually efficient mapping from problems
to parallel systems. In the case where multiple programming languages are 
required, this environment supports a seamless and modular way
to interface the data and control transfer among processes in different
languages.

With its visual programming interface, modular program structure, 
dataflow-based execution, interactive visualization functionality,
and its open system characteristics, a process and dataflow-based integration 
software like AVS provides an excellent framework with which to
facilitate the integration of the various system components required by large, 
multidisciplinary applications, including the integration of sophisticated 
interactive visualization, database management, heterogeneous networking, 
massively parallel processing, and real-time decision making. It is also
a useful tool for software development and project planning. We believe
that with the adoption of HPCC technologies into industry
applications, system integration will play an important, ever-growing 
role in the development and system design of parallel software.
