\setcounter{equation}{0}
\setcounter{figure}{0}

\chapter{Integrating Data Visualization and Parallel Computations}
\markright{CHAPTER 4. VISUALIZATION AND PARALLEL COMPUTATIONS}

Network-based concurrent computing and interactive data visualization are 
two important components in industrial applications of high-performance
computing and communication. In this chapter we apply
the proposed execution framework to build
interactive remote visualization systems for real-world applications
on heterogeneous parallel and distributed computers. Using the dataflow model of
the AVS testbed in two case studies, we demonstrate
a simple, effective, and modular approach for coupling parallel simulation
modules into an interactive remote visualization environment. The
applications described in this paper are drawn from our projects
in financial modeling and computational electromagnetics for industry.
\section{Introduction}

Scientific visualization has traditionally been carried out interactively on
workstations, or in post-processing or batch on supercomputers. With advances
in high-performance computing systems and networking technologies, interactive
visualization in a distributed environment becomes feasible. In a remote
visualization environment, data, I/O, computation and user interaction are
physically distributed through high-speed networking to achieve high
performance and optimal use of various resources required by the application
task. Seamless integration of high-performance computing systems with graphics
workstations and traditional scientific visualization is not only feasible, but
will be a common practice with real-time application systems.

Advances in parallel computing systems and network technology provide new 
opportunities for implementing computationally intensive applications.
Applications that integrate modeling and simulation 
with large-scale information processing often require an interactive graphical 
user interface (GUI) in a real-time computing environment.  Most GUIs are 
event-driven and serial in nature, making them unsuitable for parallel 
implementation.  On the other hand, visualization tools for parallel systems often 
require special hardware support or are developed for a specific 
hardware/software system and are not portable. 
Parallel computing environments of the future will likely be based 
on a heterogeneous, networked computing system and will be required to
solve applications with various requirements in
scientific computation, communication, programming models, 
memory hierarchies, I/O, database management, and visualization. 
Network-based concurrent computing and advanced data visualization are 
two important components in real-world HPDC applications,  
especially for industrial problems 
requiring real-time simulation/modeling, information processing, and 
system integration. 

In several real-world HPDC application projects
we have developed real-time parallel numerical 
simulation/modeling applications in the areas of 
financial modeling \cite{cheng1,cheng4},
computational electromagntics \cite{cheng2}, computational chemistry \cite{lin95a},
and environmental
modeling \cite{cheng3}. These applications use a range of parallel and distributed
systems from MPP machines that include the CM2, CM5, DECmpp-12000, and IBM SP2 to
networked heterogeneous workstation clusters such as the DEC Alpha Farm
and coupled MPPs and workstations. These projects are part of the major HPDC 
technology transfer program, InfoMall \cite{infomall}, developed at NPAC,
that consists of a set of interactive distributed high-performance 
information systems collectively
called InfoVision \cite{infovision} (INFOrmation, Video, Imagery, and SImulation ON-demand).
Targeted at state-of-the-art applications and enabling technology on NYNET 
(a regional ATM-based WAN network) and the utilization of the supercomputing resources at 
NPAC, InfoVision focuses on real-time information production, 
analysis, access, and integration on distributed and parallel computing systems.
This allows geographically located regional 
end-users to interactively access multimedia information and to remotely 
perform parallel simulation demonstrations on supercomputers over the 
high-speed WAN network, NYNET.
        
As a core-enabling technology in simulation on-demand applications, 
the interactive software integration environment plays a central role in coupling
parallel and distributed computing, interactive remote user-interface,
advanced data visualization, and transparent networking to support 
real-time simulations over a heterogeneous computing and networking
environment. 

In this chapter we describe our approach to building interactive 
visualization systems for the InfoVision simulation-on-demand
projects. We use AVS as the integration
framework testbed to facilitate both networking and scientific data visualization
in all the case studies. This chapter is organized as follows:  
In Section 4.2 we briefly review some related work in embedding
parallel applications within the framework of modular distributed
visualization systems. In Section 4.3
we detail a distributed visualization 
and computing framework built on the proposed dataflow model and to be elaborated on in later
sections when it is applied to specific applications and parallel systems. Section 4.4
describes a prototype visualization system for a financial option-price
modeling application originally implemented on a CM5 and a DECmpp-12000 and
ported to an IBM SP2 and a DEC Alpha farm. A remote visualization 
environment for an electromagnetic scattering simulation on
a CM5 and networked workstations is given in Section 5 to demonstrate the
effectiveness of this framework. Finally, we state our conclusions in Section 4.6.

\section{Related Work}

Research and applications in distributed visualization environments
have been very active in the HPCC community since the early 1980's. 
Interactive visualization
is identified as a critical component in the national HPCC
initiatives \cite{hpcc1,hpcc2}. 
Many efforts are being made toward developing interactive
distributed visualization applications on parallel systems 
in the framework of the AVS dataflow model 
\cite{grant,arsi,wolfgang,karen,raymond}. Most efforts, 
however, are focused on customized solutions in which front-end
visualization/graphics components are tightly coupled with
back-end parallel engines where specialized graphics hardware
is usually required to support high-end graphics capability.
To be able to support a general interactive
software development environment for HPDC applications,
our approach is focused on a loosely coupled visualization and computing
framework portable to different distributed computing system
configurations.

Similar work in this direction includes CM/AVS and MP/Express.
CM/AVS \cite{gary,cmavs}, developed by Thinking Machines, is a modular visualization
system in which AVS is used as the front-end visual
programming interface for scientific data visualization. It is
tightly coupled with the CM-5 parallel supercomputer for back-end parallel 
computations. The major
difference between CM/AVS and the distributed AVS environment is that 
instead of conventional TCP/IP sockets and UNIX-shared memory, 
specialized CM-5 shared-memory regions or CM-domain sockets are 
used in CM/AVS to exchange data among distributed parallel modules 
on CM-5 processor partitions.

MP/Express \cite{express}, a research project in AVS, Inc., is an 
enhanced AVS/Express for building parallel and distributed AVS 
applications with a visually-based software
development tool. It incorporates a middle communication layer between
the AVS kernel and application modules such that distributed modules
can communicate with each other using the portable message-passing layer and
a data layout system for automatic data distribution among multiple
node processes of a distributed module in an SPMD (Single Program Multiple Data)
paradigm. The serialized data transfer channels between parallel modules found in
AVS become parallel channels in MP/Express, and an automatic data layout for
splitting and recombining data sets in AVS data types is available to 
module developers.

\section {A Distributed Visualization and Computing Model in AVS}
\subsection{A Dataflow-Based Visualization and Computing Model}
In a typical remote visualization environment
a simulation cycle is started from a GUI module running on a local machine, 
with simulation 
parameters represented as slide buttons and dials that provide 
real-time instrumentation of the simulations' progress, as well as simulation 
steering (changing simulation parameters before a new simulation cycle).
Source data for the simulation can also be read in by the GUI module from
databases or data files. The computational task of the simulation is decomposed
into several computationally independent subtasks that are represented as
computing modules
distributed to different remote machines.
Input data are collected from the
GUI by user runtime interaction or from disk files, or from both, 
and broadcast
to computing modules on the
remote machines. 
There is little or no data
transfer among modules on different remote machines. 
A remote machine may be a workstation 
or a supercomputer, whichever architecture 
and computational power is best suited 
to the decomposed subtask. The simulation ends with some kind of 
GUI rendering/viewing modules that run on the local machine and 
use results generated from remote modules. Fundamentally, this is
a dataflow (data-driven) programming model in which the activation of a module
process is triggered solely by the availability of input data 
from another module process on either the same or a different machine. 

\begin{figure}[tb]
\centerline{\epsfxsize=450pt \epsfbox[50 430 550 720]{chapter4-visualization/framework.eps}}
%%BoundingBox: 50 420 500 730
\caption{A general visualization/computing model}
\label{framework}
\end{figure}

This general model is well suited to rapid-prototyping for certain simulation
and modeling applications that require both scientific data visualization and 
high-performance computing.
At the software environment level this model requires only the
support of high-level
data visualization and networking facilities.
Most important, this data flow-based model is well supported by most 
commercially available data visualization environments such as AVS \cite{avs1} from AVS Inc.,
Explorer \cite{iris} from Silicon Graphics, Inc., and Data Explorer from IBM.

As shown in Figure \ref{framework}, using the dataflow programming model in AVS, message passing among modules
on the same machine and different machines is identical and completely 
transparent to a module programmer. The AVS kernel (protocol) supervises 
data transfer, which is eventually carried out by TCP/IP at a lower level.
The module programmer needs only to define module input and output ports
in AVS predefined data types. Message passing among AVS modules 
occurs only through I/O ports and uses AVS's IPC or RPC protocol. 
A set of routines for initializing and 
describing modules to AVS, as well as parameter handling, accessing data, 
error handling, and
coroutine event handling, is provided. 
Data sources and destinations can flexibly be defined by 
visually connecting module input and output ports
using a network editor.

Data transfers between host and node programs of a parallel module
rely on either portable message-passing protocols in implementing
a parallel module such as PVM and MPI, or on native protocols
specific to a parallel system such as CMMD, CMFortran on a CM5, MPL on an IBM SP2,
or HPF on a DECmpp. They are strictly confined within a parallel module.
Message passing between two parallel modules on different machines must
go through the dataflow channel between the two sequential
AVS modules embedded in
the host parts of the parallel programs.

\subsection{Performance Analysis}
A generalized performance model of our networked system is shown in Figure \ref{pmodel}.
In one complete modeling cycle, starting with the broadcast of new data from 
$M_I$, and ending with renderings on $M_h$, let $t_{\rm calc}$
be the summed calculation time on all machines and let $t_{\rm comm}$ be the 
summed communication time in the system.  Let $T_d, T_s$ be the total time to 
complete such a cycle in the distributed system and in a single-machine 
system, respectively.  Other symbols are defined as follows: 
$c_{\rm k}$ = communication time of required message passing from $M_I$ to $M_k$;
$d_{\rm k}$ = communication time of required message passing from $M_k$ to $M_h$;
$m_{\rm k}$ = calculation time of performing required computation on $M_k$;
$r_{\rm k}$ = calculation time of performing required rendering by $P_k$ on $M_k$;
$t'_{\rm calc}$ = the total calculation time on a single-machine system;
$m'_{\rm k}$ = calculation time of performing required computation on 
a single machine system.

\begin{figure}
\centerline{\epsfxsize=450pt \epsfbox[50 550 550 720]{chapter4-visualization/pmodel.eps}}
%BoundingBox: 0 270 612 740
\label{pmodel}
\caption{A general performance model}
\end{figure}

\begin{figure}
\centerline{\epsfxsize=450pt \epsfbox[50 480 550 720]{chapter4-visualization/fig4.eps}}
\caption{Different cases of the pipelined calculations and communications}
\label{pipeline}
\end{figure}

In a single-machine system, $M_I = M_h = M_1 = M_2 = \cdots = M_n, 
 t_{\rm comm} = 0$ and $T_s = t'_{\rm calc} = \displaystyle{\sum^n_{k=1}}\
(m'_k+r_k)$.  In the distributed system based on TCP/IP protocol, we observe:
sequential message passing from $M_I$ to $M_k$, although the sending 
order can be scheduled;
execution of computing process on $M_k$ upon receiving a new message 
from $M_I$;
and sequential rendering of process $P_k$ on a single-processor 
machine $M_h$, where each $P_k$ is activated only after receiving a new 
message 
from $M_k$, and no other $P_j (j\ne  k)$ is running on $M_h$. 

Calculation and communication among different machines in the system 
are \\pipelined.  We can overlap the following processes: (1) $c_k$ with $m_j, 
d_j$, and $r_j$; (2) $m_k$ with $d_j$ and $r_j$; and   
(3) $d_k$ with $r_j$.  
Figure \ref{pmodel} illustrates this overlap of processes in  a
system of $(n = 2)$ machines.
For each machine $M_k$, the total time to complete a cycle is
 $T_d=c_k+m_k+d_k+r_k+i_k$ , where $i_k$ is the idle time of $M_k$ in the 
cycle, excluding the waiting time $r_k$ for the rendering on $M_h$.
Thus, 
$nT_d = t_{\rm comm} + t_{\rm calc} + t_{\rm idle}$, where
 $t_{\rm comm}=\displaystyle{\sum^n_{k=1}}(c_k+d_k)$,
 $t_{\rm calc} = \displaystyle{\sum^n_{k=1}}(m_k+r_k)$, and
 $t_{\rm idle} = \displaystyle{\sum^n_{k=1}}i_k$.

The speedup $S$ of a distributed system over a 
single-machine system is:
$$S= {T_s\over T_d} ={n\over{t_{\rm calc}\over t'_{\rm calc}}+
{t_{\rm comm}\over t'_{\rm calc}}+{t_{\rm idle}\over t'_{\rm calc}}}.$$

Speedup depends on four components: (1) $n$, the number of distributed 
computing modules; (2) $t_{\rm calc}/t'_{\rm calc}$, the ratio of total 
calculation time on the distributed system to calculation time on a 
single-machine
 system; (3) $t_{\rm comm}/t'_{calc}$, the ratio of total communication time 
in the distributed system to calculation time on a single-machine system; 
and (4)  $t_{\rm idle}/t'_{\rm calc}$, the ratio of total idle time in the 
distributed
system to calculation time on a single-machine system.  As Figure \ref{pipeline} 
illustrates, for a given application on a given system (fixed $c_{\rm k}$,
$d_{\rm k}$,$m_{\rm k}$,$r_{\rm k}$)
$t_{\rm idle}$ depends on the order of messages sent from $M_I$ to $M_k$,
making this a scheduling problem.  In the best case,
$t_{\rm idle}=\displaystyle{\sum^{n-1}_{k=1}}c_k(n-k)+ 
\displaystyle{\sum^n_{k=1}} a_kr_k,$
 where $\{a_1, a_2, \ldots, a_n\}$ is a permutation of the set $\{0,1, 
\ldots, n-1\}$.

\section {Case Study I --- An Interactive Visualization Environment for 
Financial Modeling on Heterogeneous Computing Systems}

Financial modeling represents a promising industrial application of 
high-performance computing. In previous work, parallel stock option pricing models
were developed for the Connection Machine-5 and the DECmpp-12000 \cite{fin1,fin2} and
were later ported on an IBM SP2 and a DEC Alpha cluster. These parallel models run
approximately two orders of magnitude faster than sequential models on
high-end workstations. To further develop this application in real-industry use, 
a portable, workstation-based, interactive visualization 
environment is developed for a heterogeneous computing environment. 

Stock option pricing models are used to calculate a price for an option
contract based on a set of market variables, (e.g., exercise price, risk-free
rate, time to maturity) and a set of model parameters. Model price estimates
are highly sensitive to parameter values for volatility of stock price,
variance of volatility, and correlation between volatility and stock price.
These model parameters are not directly observable and must be estimated from
market data. Using optimization techniques for model parameter estimation holds
great promise for improving model accuracy.

We use a set of four option pricing models in this study. Simple models treat
stock price volatility as a constant, and they price only European (option exercised
only at maturity of contract) options. More sophisticated models incorporate
stochastic volatility processes and price American contracts (option exercised
at any time in life of contract) \cite{option1,option2}. These models are computationally
intensive and have significant communication requirements. 

Analytic models are useful tools in the financial market, but require expert
interpretation. To further evaluate and optimize pricing models to run in a
realistic application environment, we combine high-performance computing modules
for real-time pricing with real-time visualization of model results and market
conditions, and a graphical user interface allowing expert interaction with
pricing models. We envision a market expert using such a system to start and
stop a set of models, adjust model parameters, and call optimization routines
according to dynamically changing market conditions.

A detailed description of this case study can be
found in Appendix C.1.

\section {Case Study II --- An Interactive Remote Visualization Environment 
for an Electromagnetic Scattering Simulation
on a High-Performance Computing System}

Electromagnetic scattering (EMS) simulation is an important
computationally intensive application within the field of electromagnetics
and is an area of emphasis
in the national high-performance computing initiative \cite {Fox}. 
Advances in high-performance computing and the data 
visualization environment provide new opportunities to visualize 
real-time simulation problems, such as EMS, that require significant 
computational resources.
In this case study an integrated interactive visualization
environment was created for an EMS simulation, coupling a graphical 
user interface (GUI) for runtime simulation parameters input
and 3D rendering output on a graphical 
workstation with computational modules running on
a parallel supercomputer and two workstations. 
The AVS dataflow integration model was used as integrating software
to facilitate both networking and scientific data visualization. 
We explore the dataflow methodology to
naturally integrate data visualization, parallel systems, and
heterogeneous computing. Major issues in integrating this remote 
visualization system are discussed, including task decomposition, 
system integration, and concurrent control --- all under a high-level 
distributed programming model.  

We describe the complete development and prototype system of this application 
in Appendix C.2.

\section {Conclusion}
High-speed networks, combined with advances in 
computational power, will support a new set of applications
designed for high-performance distributed computing.
In real-world HPCC applications, computational power alone
is not sufficient. We see a need for a broader parallel
software engineering environment, such as the interactive
visualization system described in this chapter, which combines and takes
advantage of high-performance computing and other 
well-established computing areas such as data visualization,
computer graphics, system integration, and software engineering. 
We focus here on an execution model
required to combine real-time modeling/simulation, parallel processing, 
visualization, and interactive user control.

With its visual programming interface, modular program structure, 
dataflow-based execution, interactive visualization functionality,
and its open system characteristics, a process-based integration framework 
like AVS provides an excellent framework with which to
facilitate integration of various system components into a large, 
multi-disciplinary application, including the integration of sophisticated 
interactive visualization, database management, heterogeneous networking, 
massively parallel processing, and real-time decision making. It is also
a useful tool for software development. We believe
that with the adoption of HPCC technologies into industry
applications, data visualization and software engineering will play an 
important ever-growing role in technology transfer and 
parallel software development.

