\setcounter{equation}{0}
\setcounter{figure}{0}

\chapter{Case Studies of Distributed Data Visualization and
Parallel Processing in the Software
Integration Framework}
\markright{APPENDIX C. CASE STUDIES IN CHAPTER 4}

This appendix details two application case studies used in
Chapter 4 to demonstrate the integration of interactive
distributed data visualization and parallel simulation
in a unified dataflow-based programming framework.
The two case studies are chosen from financial modeling and
computational electromagnetics. For
each application we describe a prototype system that illustrates
system configuration, software integration,
performance, functionality and other issues. 

\section {A Financial Modeling Application}
\subsection{Problem Description}

Stock options are contracts that give the holder of the contract the right 
to buy or sell the underlying stock at some time in the future.  
Option contracts are traded 
just as stocks are traded, and models that quickly and accurately price option 
contracts are valuable to traders and financial managers.  Speculators 
participate in the option market to capture potentially high profits with 
relatively small investment capital.  Financial managers buy and sell options 
to hedge risk in their investment portfolios.

We use a set of four option pricing models in this study. Simple models treat
stock price volatility as a constant, and price only European (option exercised
only at maturity of contract) options. More sophisticated models incorporate
stochastic volatility processes, and price American contracts (option exercised
at any time in life of contract) \cite{option1,option2}. These models are computationally
intensive and have significant communication requirements. The four pricing
models are: BS --- the Black-Scholes constant volatility, European model; AMC ---
the American binomial, constant volatility model; EUS --- the European binomial,
stochastic volatility model; and AMS --- the American binomial, stochastic
volatility model. Detailed descriptions of these four models can be found
in \cite{option1,fin1}.

\subsection{System Configuration and Integration}

Using the stock option-price modeling application as a case study, we demonstrate
a simple, effective, and modular approach to coupling network-based concurrent
modules into an interactive remote visualization environment as proposed in Chapter 4.3.1. 
Two prototype simulation on-demand systems are developed to demonstrate the portability
of this framework in which parallel option pricing
models are locally implemented on two system configurations (two meta machines):
one with two MPP machines --- a 32-node CM5 and a 8K-node DECmpp-12000;
the other with two distributed systems --- an Ethernet-based IBM SP2 and 
a cluster of DEC Alpha workstations networked by a
FDDI-based GIGAswitch. They are coupled with an
interactive graphical user interface over the NYNET ATM-based wide area
network. AVS is used to integrate massively parallel processing, workstation-based
visualization, an interactive system control, and distributed I/O modules.

\subsection*{Configuration 1 --- NYNET + CM5 + DECmpp-12000 + Workstations}

Figure \ref{config3} is the system configuration of the first prototype interactive
simulation-on-demand system for the option price modeling application,
utilizing the network infrastructure and
supercomputing facility at NPAC.

\begin{figure}
\centerline{\epsfxsize=435pt \epsfbox[50 75 550 710]{chapter4-visualization/fig1.1.eps}}
%BoundingBox: 0 0 612 725
\caption{System Configuration 1 for the Financial Modeling on a CM-5 and a DECmpp-12000}
\label{config3}
\end{figure}

The AVS kernel runs on a SUN10 workstation, which acts both as an AVS server to
coordinate data-flow and top-level concurrent control among remote modules and
as a network gateway that links the NPAC in-house host machines locally
networked by an Ethernet to the regional end-user through the NYNET. The
ATM-based link is built around two Fore switches that operate at 155 Mbps
(OC3c) while the wide-area network portion of the network operates at a speed of
OC48(2400 Mbps).

Our heterogeneous computing system for stock option pricing consists of four
compute nodes, a home machine, and two file server machines. All workstations,
including the front-ends of the DECmpp-12000 and CM-5, are connected by a
10MBit/second Ethernet-based LAN.

The four option pricing models run on remote compute nodes: BS model on a
DEC5000, AMC model on a SUN4, EUS model on a CM-5 and AMS on a
DECmpp-12000(SX). Each remote compute node has its own I/O capability. Our
DECmpp-12000 is a massively parallel SIMD system with 8192 processors. Each
RISC-like processor has a control processor, 40 32-bit registers, and 16
KBytes of RAM. All the processor elements are arranged in a rectangular
two-dimensional grid and are tightly coupled with a DEC5000 front-end
workstation. The theoretical peak performance is 650 Mflops DP. Our CM-5 is a
parallel MIMD machine with 32 processing nodes. Each processing node consists
of a SPARC processor for control, four proprietary vector units for numerical
computation, and 32 MBytes of RAM. The control node of the CM-5 is a SUN4
workstation. The theoretical peak performance is 4 Gflops. Sequential compute
nodes include a DEC5000 and a SUN4. The DEC5000 performs at 6.8 Mflops, and has
16 Mbytes memory. The SUN4 runs at 4.3 Mflops and has 32 Mbytes memory.

While displayed on the end-user's home machine, a user interface actually runs
on a remote SUN4 that combines user runtime input (model parameters, network
configuration) with historical market databases stored on disk, and broadcasts
this data to remote compute nodes. Top-level system synchronization occurs with
each broadcast.

An IBM RS/6000 is used as a file server for non graphical output of model data.
In this application, model prices calculated at remote compute nodes, along
with market data, are written to a database on the output file server 
for later analysis.

In summary, the heterogeneous computing system illustrated in Figure \ref{config3} provides
distributed computing, distributed memory, and distributed input/output for the
stock option pricing application.

Our heterogeneous computing system integrates diverse functions, computation,
visualization, and system control over a diverse set of hardware. We use a mix
of programming languages on the remote compute nodes-Fortran77 on the DEC5000,
C on the SUN4, CMFortran on the CM-5, and MPL (data parallel C) on the
DECmpp-12000. At the operating system level, all remote modules are compiled and
linked as stand-alone programs. Input and output ports are defined in modules
by the programmer using specific library routines provided by AVS. Each module
represents a process. 

There are two sources of input data: historical market data read from disk
files, and runtime input of model parameters by the user through a GUI on the
home machine. Output
from all four models is rendered in a graphics window of the GUI and displayed numerically in
a shell window, both on the home machine, and written to a database on the file server.

\begin{figure}
\centerline{\psfig{figure=chapter4-visualization/fingui.ps,height=3.0in,width=5.5in}}
%width=6.0in}}
%BoundingBox: 0 0 612 728
\caption{The graphical user interface for the option price modeling (1)}
\label{gui3}
\end{figure}

Figure \ref{gui3} illustrates the GUI for managing user runtime input and output, and
the system configuration. Runtime input includes user-defined model parameters
and system execution modes (continuous, step or pause). 
Outputs include two-dimensional displays of model
and market prices calculated by the compute nodes. The system configuration
includes choice of pricing models, network configurations, and interface
layouts.

Pricing models are extremely sensitive to model parameters for implied
volatility, variance of stock volatility and correlation between stock price
and volatility. These parameters may be read from data files (historical
estimates), calculated just prior to running the pricing models (by turning on
the optimization button from the GUI), or defined at run time (by expert user's
parameter steering from the GUI).


\subsection*{Configuration 2 --- NYNET + IBM SP2 + DEC Alpha Farm + Workstations}
Figure \ref{config4} is the system configuration of the second prototype using the same
visualization/computing framework while the two parallel pricing models
now ported to two clusters instead of the two MPPs in the previous
configuration.

\begin{figure}
\centerline{\epsfxsize=435pt \epsfbox[50 75 550 710]{chapter4-visualization/fig1.2.eps}}
%BoundingBox: 0 0 612 725
\caption{System Configuration 2 for the Financial Modeling 
on an IBM SP2 and a DEC Alpha Farm over NYNET}
\label{config4}
\end{figure}

The two parallel pricing models (EUS model and AMS model) are implemented in
PVM and run, respectively, on an 8-node IBM SP2, networked by an Ethernet at the
time of evaluation, and an 8-node DEC Alpha cluster interconnected by an
FDDI-based GIGAswitch. They are coupled under the proposed AVS environment with
the other two sequential simple models (BS model and AMC model) running on a
SUN4 and a DEC5000 workstation, respectively. The nodal processor of SP2 is an IBM
RISC/6000 processor running at 62.5 MHz.
The DEC Alpha farm consists of eight Alpha model 4000
workstations that are supported by a high-performance networking backbone of 
dedicated, switched FDDI segments. The GIGAswitch provides full FDDI bandwidth
and low latency switching to every workstation in the farm.

All models output is graphically displayed on the end-user's home machine (a
SUN10) in AVS graph viewers. Figure \ref{gui4} is the screen dump of the GUI which is the same
as that in Figure 2 with a different layout that shows the
simulation control panel (left), model output windows (top), and the flow
network (bottom).

\begin{figure}
\centerline{\epsfxsize=450pt \epsfbox[95 370 517 773]{chapter4-visualization/fig2.ps}}
%BoundingBox:  95 370 517 773
\caption{The graphical user interface for the option price modeling (2)}
\label{gui4}
\end{figure}

\subsection{Performance Analysis}

Timings observed for the system configuration 
in Figure \ref{config3} are listed in Table C.7.  According to our analysis in Chapter 4.3.2, 
the expected
$T_d = c_1 + m_1 + d_1 + r_1 + r_2 + r_3 + r_4  = 0.017 \, +\, 0.015\, +\, 
0.1\, + \, 4^*0.9 = 3.642$ seconds (assuming the scheduled order is
$c_1, c_2, c_3$, and  $c_4$).  This value will vary because the system 
runs under a resource-and-time-sharing environment.

To compare the performance of the distributed configuration and a 
single-machine system, 
we also list timings from a SUN4 workstation in Table C.7. 
Thus, $T_s = 0.015\, +\, 0.085\, +\, 4.05\, +\, 4.25\, +\, 4^*0.9 = 12.0$,
and we calculate an expected speed-up of $S = 12.0/3.642 = 3.3$.  

\begin{table}
\begin{center}
\begin{tabular}{|l|c|c|c|c|} \hline
price model/machine & $m_k$ & $c_k$ 
& $d_k$ & $r_k$ \\
&     &$(M_I\Rightarrow M_k)$&($M_k\Rightarrow M_h)$&\\\hline\hline
{\bf BS}/DEC5000($M_1$)          & 0.015 & 0.017 & 0.1 & 0.9 \\\hline
{\bf AMC}/SUN4($M_2$)            & 0.085 & 0.017 & 0.1 & 0.9 \\\hline
{\bf EUS}/8K DECmpp 12000($M_3$) & 0.075 & 0.017 & 0.1 & 0.9 \\
{\bf EUS}/SUN4($M_3$)            & 4.05  &       &     &     \\\hline
{\bf AMS}/32-node CM5($M_4$)     & 0.025 & 0.017 & 0.1 & 0.9 \\
{\bf AMS}/8K DECmpp 12000($M_4$) & 0.045 &       &     &     \\
{\bf AMS}/SUN4($M_4$)            & 4.25  &       &     &     \\\hline
\end{tabular}
\end{center}
\caption{Experimental timings for the option price modeling (in seconds)}
\label{fintiming}
\end{table}

\section {A Computational Electromagnetic Application}
\subsection{Problem Description}

Electromagnetic scattering is a widely encountered problem 
in electromagnetics \cite{Har1,Jordon,Van}, with important applications in 
industry such as microwave equipment, 
radar, antenna, aviation, and electromagnetic compatibility design.
Figure \ref{ems} illustrates the EMS problem we are modeling. 
Above an infinite conductor plane there is an incident EM field in 
free space. Two slots of equal width on the conducting plane 
are interconnected to a microwave network behind the plane. 
The microwave network represents the load of waveguides --- 
for example, a microwave receiver. The incident EM field penetrates
the two slots which are filled with insulation materials such as air or oil. 
Connected by the
microwave network, the EM fields in the two slots interact with each other, 
creating two equivalent magnetic current sources in the two
slots. 
A new scattered EM field is then
formed above the slots. We simulate
this phenomena and calculate the strength of the scattered EM 
field under various physical circumstances.
The presence of the two slots and the microwave load in this application 
requires simulation models with
high-performance computation and communication.
Visualization is very 
important in helping scientists to
understand this problem under various physical conditions.

\setcounter{figure}{5}
\begin{figure}
%\centerline{\psfig{figure=chapter4-visualization/fig4-5.eps,height=6.0in,width=5.0in}}
\centerline{\epsfxsize=370pt \epsfbox[80 220 550 720]{chapter4-visualization/fig4-5.eps}}
%BoundingBox: 50 220 500 715
\caption{Physical parameters of the EM scattering problem}
\label{ems}
\end{figure}


In previous work, data-parallel and message-passing algorithms for this 
application were developed to run efficiently on
massively parallel SIMD machines such as the Connection Machine CM-2 and the
DECmpp-12000, and MIMD machines such as the
Connection Machine CM-5 and the iPSC/860. The data-parallel algorithms 
run approximately 400 times faster than 
sequential versions on a high-speed workstation \cite{Lu3}. 
Parallel models on high performance systems 
provide a unique opportunity to interactively visualize the EMS
simulation in real time. This problem requires 
a response time of a simulation cycle that is not possible
on conventional hardware.

Figure \ref{ems} also shows the physical parameters of the electromagnetic scattering
problem.

\subsection{Parallization of the EMS Simulation Model}

Components of our distributed model for simulation and visualization are:
\begin{enumerate}
\item Decomposition of visualization computation
\item Selection of simulation parameters
\item Design of a graphical user interface
\item System integration and synchronization
\item Interstage communication and performance analysis
\end{enumerate}

Using the EMS simulation, we next examine the decomposition of visualization 
computation and the selection of simulation parameters. 
The remaining components are discussed in later sections.
 
\subsubsection{Computational Task Decomposition}

Control parallelism and data parallelism are used to target the distributed 
computing model illustrated in the preceding 
section and an MIMD parallel machine, along with a number of
high-speed workstations.
We use the following guidelines for the decomposition of our model's
control parallelism:

\begin{enumerate}
\item Hardware architecture and computational power best suited  to
decomposed subtasks
\item Logical components of the application's computational model
\item Performance balance among decomposed modules and
\item Communication requirements
\end{enumerate}

The moment method \cite{Har2,Har4,Lu2} is used as the numerical
model for the EMS problem, which can be
represented as: 
\[
\{\left [ \begin{array}{c} Y^{a} \end{array} \right ] + \left [ \begin{array}
{c} Y^{b} \end{array} \right ] \} \vec{V} = \vec{I}
\]

\[
\left [ \begin{array}{c} H \end{array} \right ] = {\cal L} \{ f~(\vec{V}, \vec{M}, 
\left [ \begin{array}{c} H_{0}^{2} \end{array} \right ] ) \}
\]  
where

$\left[\begin{array}{c} Y^{a} \end{array}\right]$ : equivalent admittance matrix of the free space;

$\left [\begin{array}{c} Y^{b} \end{array}\right]$ : equivalent admittance
matrix of the microwave network;

$\vec{V}$ : coefficient vector; 

$\vec{I}$ : the excitation vector; 

$ \vec{M}$ : a vector of mode functions;

$\left[\begin{array}{c} H_{0}^{2} \end{array}\right]$ : matrix of Hankel functions;

$f$ : a function; 

$ \cal L$ : a linear operator on f;

$\left [\begin{array}{c} H \end{array}\right ]$ : final matrix of the simulated EMS field strength.\\


From the previous parallel algorithm design, we observed that:

\begin{enumerate}
\item Calculations of $\left [\begin{array}{c} Y^{a} \end{array}\right ]$, $\left [\begin{array}{c} Y^{b} \end{array}\right ]$, $\vec{I}$, $ \vec{M}$,  and $\left [ \begin{array}{c} H_{0}^{2} \end{array}\right ]$  can be done independently.
\item Computation of $\left [\begin{array}{c} Y^{a} \end{array}\right ]$, $\left [ \begin{array}{c} H_{0}^{2} \end{array}\right ]$, and the linear solver for $\vec{V}$ have
significant communication requirements and are computationally intensive.
\item $\left [\begin{array}{c} Y^{b} \end{array}\right ]$ is a sparse matrix and calculation of $\vec{M}$  requires little time.
Calculation time for $\left [\begin{array}{c} Y^{a} \end{array}\right ]$, $\left [\begin{array}{c} Y^{b} \end{array}\right ]$  and $\vec{I}$  are relatively balanced.\\
\end{enumerate}

Thus we partition computations of this application into four 
loosely coupled computing modules (called EM-1-SUN, EM-2-SUN,
EM-3-CM5 and EM-all-CM5 in the AVS module network of Figure \ref{gui6}). 
Three modules can run simultaneously in the distributed computing 
environment (see Figure \ref{config6}).

At the second level, i.e., data decomposition, because most computations
of $\left [\begin{array}{c} Y^{a} \end{array} \right ]$, $\vec{V}$(a linear solver), $\left [ \begin{array}{c} H_{0}^{2} \end{array}\right ]$, and $\left [ \begin{array}{c} H \end{array}\right ]$ are matrix manipulations, 
data parallel algorithms are developed in Fortran90 and tailored to run on 
the Connection Machine 5 to take advantage of the CM5's balanced data 
network and control network. The CM Scientific Subroutine Library
(CMSSL) is used in the data-parallel implementation \cite{Lu3}.

At a more general level, we can view the entire system as a ``metacomputer'' that makes use of both 
functional parallelism and pipelining. In this application,
functional parallelism consists of graphical I/O (i.e., user interaction, 3D rendering)
and decomposed simulation computations that are handled concurrently by different components
of the metacomputer. Pipelining combines calculations and communications among 
different processors or groups of processors (e.g. the CM5) that 
are carried out  simultaneously in consecutive stages of the simulation. 
We will discuss pipelining execution of the system in later sections. 

\subsubsection{Selection of Simulation Parameters}

Simulation parameters are implemented as control widgets within a graphical
user interface 
to provide a visual medium that allows the user to 
interact with the simulation and visualization at runtime.
We choose parameters representing the physical 
foundation of the EMS problem and
the computational model that the user can manipulate in order to 
visually understand the problem.

As shown in Figure \ref{ems}, the following parameters are graphically represented to allow
the user's runtime input:
\begin{enumerate}
\item $\Phi$ : angle of the incident EM field(wave)
\item $i$ : $i \times i$ is the simulated area of the EMS field
\item $w$ : width of the slot
\item $d$ : distance between the two slots
\item $\epsilon$ : permittivity of the media in the slot
\item $\mu$ : permeability of the media in the slot
\item $h$ : height of the slots
\item $y_{11}, y_{12}, y_{21}, y_{22}$ : admittance parameters of a given microwave network
\end{enumerate}

In addition, four parameters represent the 
characteristics of the equivalent magnetic current originally
formed by the  penetrating incident EM field. 
These parameters are used as mode function expansions and the number of pulse 
functions in the moment method. Another parameter used for visualization purposes is
the number of grid points for discretizing the visualized EMS area.
All five parameters have direct impact on 
the computational requirements and simulation resolution of the moment
method. Using different combinations of the parameters, we can visualize
the EMS simulation within a large number of real application 
circumstances. We also use visualization of typical parameters
to verify the EMS computational model 
and some well-known theories about this physical phenomena.

\begin{figure}
\centerline{\epsfxsize=430pt \epsfbox[80 280 550 720]{chapter4-visualization/fig3.eps}}
%BoundingBox: 0 300 612 740
\caption{The System configuration for the EMS application}
\label{config6}
\end{figure}

\subsection{Implementation of the Remote Visualization Environment}

\subsubsection{System Configuration and Integration Using AVS}

Figure \ref{config6} illustrates the system configuration and module
components
distributed over the network to three high-end workstations and a supercomputer
Connection Machine 5. The network is a
ten MBit/s Ethernet-based local network. 
AVS is used to 
provide the sophisticated 3D data visualization and system control
functionality required by the simulation.
We use AVS to facilitate high-level networking and data transfers among 
visualization and computational modules on different machines in the system.
AVS provides a 
data-channel abstraction that transparently handles
type-conversion and module connectivities. This software system is optimized for data
movement by using techniques such as shared-memory message passing among modules
on the same machine. 
Message passing occurs at a high level of data abstraction in AVS. This approach helps to
make optimal use of
both the high-performance computing resources and the rendering capabilities 
of the local graphical workstation.
The transparent networking capabilities of AVS open up possibilities for 
visualization far beyond traditional graphics capabilities \cite{avs1,avs2}. 

The local machine in our system is an IBM RS/6000 with a 24-bit 
color GTO graphics adaptor. 
An AVS coroutine module (in C) on the local machine 
serves as a graphical input and system control interface to monitor
and collect user runtime interaction with the simulation 
through keyboard, mouse and other I/O devices. The AVS kernel 
also runs on the local machine, coordinating 
data flows and control flows among AVS (remote) modules in the network.

We use an AVS system module called ``geometry viewer''
along  with other system modules (``generate colormap,'' ``color range,'' and
``field to mesh'') for 3D rendering operations.

The computationally intensive modules of this application are distributed
to the CM5 \cite{cm5}, an MIMD supercomputer that is configured 
with 32 processing nodes at NPAC. 
Each processing node (PN)
of the CM5 consists of a SPARC processor for control and non-vector 
computation, four vector units for numerical computation, and 32 MB of RAM.
It also includes a Network Interface chip that gives the node access to the 
CM5 internal Data Network and Control Network. The two internal 
networks connect all the PNs with
a control 
processor (CP) that runs a custom version of SunOS on a SPARC host. 

Two Sun SPARC workstations are used in our distributed visualization 
environment to run the computational modules 
with modest communication requirements. 

All modules other than those on the local machine are 
implemented as AVS remote modules. Their input/output ports are defined by 
specific AVS libraries for receiving/sending data from/to other (remote) 
modules via socket connections.
This configuration allows the interrupt-driven user
interface input mechanisms and rendering operations to be relegated to 
the graphical workstation, while the computationally intensive components 
run on the CM5 coupled with the two
workstations. This distributed simulation environment implemented in
AVS provides a transparent mechanism for using distributed 
computing resources along with a sophisticated user 
interface component that permits
a variety of interactive, application-specified inputs.

The flow-chart diagram in Figure \ref{gui6} (lower right) is 
a module network configured by the AVS Network 
Editor for this application. 

\begin{figure}
%\centerline{\psfig{figure=chapter4-visualization/fig6.ps,width=4.4in,height=5.60in}}
\centerline{\epsfxsize=300pt \epsfbox[125 150 490 610]{chapter4-visualization/fig6.ps}}
%BoundingBox: 125 150 490 610
\caption{A Graphic user interface for the EMS visualization system}
\label{gui6}
\end{figure}


\subsubsection{The Graphical User Interface}

The graphical user interface includes a main control panel, three individual 
input panels, and a 3D rendering window. 

The main control panel provides the user with parameters input and simulation
control at runtime. There are seven dial widgets representing
the simulation parameters 
used by all computing modules on the three remote machines, and a 
control button for starting a new simulation cycle(see lower left in Figure \ref{gui6}).
The rest of the simulation parameters discussed in 
Section C.2.2 are implemented as dial widgets in individual panels 
associated  with only the modules requiring them
to minimize redundant data transfers
between the input interface module on the local machine and computational
modules on remote machines. They can also be 
turned on from the AVS Network Control Panel (not shown in Figure \ref{gui6}).

Using the AVS Geometry Viewer, 3D simulation data can be rendered in various
forms such as move, rotate, scale, move-the-eye-point and perspective view,
and with sophisticated rendering techniques such as lighting and shading, multiple camera,
Z-buffering, 2D and 3D texture mapping, automatic removal of hidden surfaces, sphere
rendering, etc. The geometry viewer also takes 
advantage of the hardware rendering capabilities of the GL 
library and the GTO graphics adaptor of the IBM RS/6000.
Figure \ref{gui6} (upper) shows a typical AVS Geometry Viewer window with two cameras from two different
angles. 

In addition, the AVS Network Editor provides a visual programming interface 
enabling the user to interactively reconfigure and reuse
network modules.
The Layout Editor in AVS allows easy and complete
customization of the control panels in the GUI.
Figure \ref{gui6} illustrates this interface.

\subsection{Interstage Communication and Synchronization}

Using the data-flow programming model in AVS, message passing is identical among modules
on the same machine and on different machines and is completely 
transparent to the module programmer. The AVS kernel (protocol) supervises 
data transfer, which is eventually carried out by TCP/IP at a lower level.
The module programmer needs only to define module input and output ports
in AVS predefined data types. Message passing among AVS modules 
occurs only through I/O ports. A set of routines for initializing and 
describing modules to AVS, as well as parameter handling, accessing data, error handling, and
coroutine event handling are provided. 
Data sources and destinations can be flexibly defined by 
visually connecting module input and output ports
using the Network Editor.

In most cases, we use an AVS ``field'' as the transferred data type,
which is actually a C structure.
In the module network shown in Figure \ref{gui6}, 
data transmissions are overlapped with simulation computations (pipelining).
For example, the computation of an {\sl EM-3-CM5} can be pipelined with the data
transfer between {\sl Input-interface-IBM} and {\sl EM-2-SUN} or {\sl EM-1-SUN}.  
Message passing within a machine is implemented
in AVS by copying pointers to the same 
memory (shared-memory). Thus
there is no network communication cost for transferring the 
matrix $\left [\begin{array}{c} Y^{b} \end{array}\right]$ between 
the two modules, i.e., {\sl EM-3-CM5} to {\sl EM-all-CM5}, on CM5's control
processor. Instead of generating a complete 3D data on the CM5 and
sending them to rendering modules on the local machine, we use a computing
module {\sl EM-3D-IBM} on the local machine to generate the $X$-$Y$ coordinate data. 
Only the Z-coordinate data (i.e., $\left [\begin{array}{c}H\end{array}\right]$) 
and two scalar parameters for defining the $X$-$Y$ data 
are transferred from the CM-5 to the local machine.
							
Simulation parameters are defined as input ports and connected
to widgets on control panels supervised by the X-window manager
on the local machine. By distributing parameters only to the modules 
that require them, we minimize  recomputation of modules in a new
simulation cycle. For example, changing of a parameter on the control panel of 
{\sl EM-1-SUN} will not activate the {\sl EM-3-CM5} and {\sl EM-2-SUN}.

This system is designed to work under a complete resource and time sharing 
environment (shared Ethernet, CM5 and all remote workstations), thus
wall-clock time for
computation and communication of all modules is difficult to predict. 
Concurrent control in the module network plays an extremely important role 
in assuring correctness, robustness and reliability. 
We issue concurrent controls in three different places:
\begin{enumerate}
\item In the main control panel of the GUI a one-shot control button is set to allow
the end-user to control the start of a new simulation cycle.
\item Broadcast of parameters from the module {\sl input-interface-IBM} to the
computing modules on the remote machines is performed only when
all the other module processes in the network are inactive.
\item Module {\sl EM-all-CM5} is implemented as a coroutine such that computations of
the linear solver for $\vec{V}$ and $\left [\begin{array}{c} H \end{array}\right ]$ will not be 
activated until all the required
input data for $\left [\begin{array}{c} Y^{a}\end{array} \right ]$, $\left [\begin{array}{c} Y^{b}\end{array} \right ]$ and $\vec{I}$ have been recieved
($\left [\begin{array}{c} Y^{a}\end{array} \right ]$ 
if only parameters on control panel 
of the {\sl EM-1-SUN} are changed).
\end{enumerate}


\subsection{Performance Analysis}
Our experiments show that under a typical working environment (only 0.5 MBits/s
of the Ethernet's ten MBits/s capacity are available), a complete simulation
cycle takes about eight seconds. This response time is quite satisfactory for this
application. Table 2 lists the timing data of major system
components. For comparison, timings of sequential implementation on a SUN4
workstation of the two parallel modules are also given in the table.

\begin{table}
\begin{center}
\begin{tabular}{|c|c|c|c|c|} \hline
           & Calculation &    Comm. time w/   & Comm. time w/ & Comm. time w/ \\   Module  &    time     & 'Input-interface-IBM &'EM-all-CM5' & 'EM-3D-IBM'   \\\hline\hline
EM-1-SUN   & 0.1(Sun)    & 0.02 & 0.045 &  n/a\\\hline
EM-2-SUN   & 0.6(Sun)    & 0.02 & 0.5   &  n/a\\\hline
EM-3-CM5   & 1.8(CM5)    & 0.02 & 0.0   &  n/a\\
           & 1260(Sun)   &      &       &     \\\hline
EM-all-CM5 & 2.1(CM5)    &  n/a &  n/a  & 0.5 \\
           & 120(Sun)    &      &       &     \\\hline
Rendering & 3.5 &  n/a &  n/a  & n/a \\\hline
\end{tabular}
\end{center}
\caption{Timings of calculations and communications for the EMS (in seconds)}
\label{emstiming}
\end{table}
