A Dataflow-Based Software Integration Model

in Parallel and Distributed Computing and Applications

by Gang Cheng

B.S., Huazhong University of Science and Technology, 1983, China M.S., Huazhong University of Science and Technology, 1988, China

Dissertation

Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the School of Computer and Information Science

in the Graduate School of Syracuse University

December 1996

Approved Professor Geoffrey C. Fox

Date \begin{matrix}c \\ ◯ \end{matrix} Copyright 1997

Gang Cheng

A Dataflow-Based Software Integration Model in Parallel and Distributed Computing and Applications

by Gang Cheng

Abstract of Dissertation

As an important component in parallel software engineering, software integration plays an indispensable role in building large-scale, multi-disciplinary parallel and distributed applications on high-performance computing systems. In this study we propose a software integration framework that is based on a dataflow programming paradigm and execution model for general-purpose HPDC applications. Advantages of this dataflow-based model are investigated and demonstrated. We study applications of this model in several specific, yet typical, areas that have general needs for parallel software integration and are well suited to this dataflow model. Drawing from extensive software and system integration case studies in those areas, we present the general requirements for a software integration framework in an HPDC application. These areas include:

1. integration of multiple parallel programming paradigms in a single unified environment;

2. integration of interactive data visualization and parallel computations on a hetergeneous distributed environment;

3. integration of networking navigation and parallel relational database management system (RDBMS); and

4. integration of World Wide Web computing and information processing.

By viewing different parallel programming paradigms as an essentially heterogeneous approach in mapping “real-world” problems to parallel systems, we discuss methodologies in integrating multiple programming models on a massively parallel system. Three applications in earth science, financial modeling, and computational electromagnetics are studied to demonstrate the integration of data-parallel and message-passing modules in the proposed multi-paradigm programming environment.

We develop interactive visualization systems on heterogeneous parallel and distributed computers. Using the dataflow model of an AVS testbed in two case studies in financial modeling and computational electromagnetics applications, we demonstrate a modular approach to couple parallel simulation modules into an interactive remote visualization environment.

Viewing the client-server model as an essentially restricted dataflow model, we describe the concept of integrating distributed computing in a World Wide Web (WWW) environment for data-intensive Internet applications. We examine approaches in integrating parallel relational databases and a Web-based networking navigation interface. Realization of the proposed dataflow integration model in Java and WWW is also described.

Acknowledgements

I am grateful to Prof. Geoffrey Fox for his continuous support as my academic advisor and NPAC director throughout this work. This thesis would not have been possible without his generosity in allowing me to continue my thesis work while working at NPAC.

I would like to thank Dr. Marek Podgorny for introducing me to parallel RDBMS and for many useful discussions, and Dr. Kim Mills for being the first to give me an opportunity to work in parallel applications.

Thanks to my colleagues, Dr. Yinhua Lu, Mr. Jay Mortensen, Dr. Ken Hawick, and Dr. Shennon Shen for help on the computational electromagnetics projects, to Piotr Sokolowski for help on the Web search project, to Dr. Wojtek Furmanski for his idea of WWWVM and WebFlow, and to Dr. Nancy McCracken for valuable advice and useful discussions. I am also indebted to NPAC for providing me with excellent computer infrastructure and systems support for this work.

I especially want to thank my uncle, John Wu, who partially sponsored my entering the United States. Without his faithful support I would not have been able to pursue a higher academic degree. I also thank Dr. Ann Shelly and Dr. Ernie Sibert, who assisted me in obtaining the precious initial scholarship from the University, and my friends, Dr. Xiangdong Yu, Dr. Xin Yi, Ms. Fang Fang, and Dr. Kang Cheng for their spiritual and moral support.

Special thanks to Ms. Elaine Weinman for editorial assistance. Finally, my greatest thanks go to my lovely wife, Yang Meng, for her patience, understanding, and sacrifice.

Contents

Acknowledgements vi

List of Tables xii

List of Figures xiii

1 Introduction 1 1.1 Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Meta-Problems and Meta-Computers . . . . . . . . . . . . . . . . . . 3 1.3 Software Integration for Meta-Problems and Meta-Computers …. 12 1.4 Software Integration in a Problem-Solving Environment . . . . . . . . 15

2 A General Software Integration Framework 20

2.1 Introduction to Dataflow Programming . . . . . . . . . . . . . . . . . 21 2.1.1 Dataflow Model As a General Programming Paradigm …. 22 2.1.2 Dataflow As the Overall Program Execution Model . . . . . . 26

2.2 A General Integration Framework Based on a Dataflow Model …. 27 2.3 Similar Work in Software Integration on HPDC Systems . . . . . . . 39 2.4 Application Visualization System — A Testbed for Software Integra-

tion in HPDC Applications . . . . . . . . . . . . . . . . . . . . . . . . 44

3 Integrating Multiple Parallel Programming Paradigms 46 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.2 Multiparadigm Parallel Programming Environments . . . . . . . . . . 48

3.2.1 Multi-paradigm Programming in Mapping a Problem Domain

to a Parallel System . . . . . . . . . . . . . . . . . . . . . . . 48 3.2.2 Multiparadigm Programming . . . . . . . . . . . . . . . . . . 52

3.3 A Dataflow-Based Integration Approach for Multiparadigm Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.3.1 A General Parallel Programming Environment on a CM5 with

AVS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 3.3.2 Integration of PVM Programs into AVS Environment . . . . . 60 3.3.3 Limitations in the AVS Integration Model . . . . . . . . . . . 62

3.4 Case Study I — Comparison of Numerical Advection Models . . . . . 65 3.5 Case Study II — Option Price Modeling . . . . . . . . . . . . . . . . 65 3.6 Case Study III — Distributed Computational Electromagnetics Systems 66 3.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

4 Integrating Data Visualization and Parallel Computations 68 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 4.3 A Distributed Visualization and Computing Model in AVS . . . . . . 72

4.3.1 A Dataflow-Based Visualization and Computing Model …. 72 4.3.2 Performance Analysis . . . . . . . . . . . . . . . . . . . . . . . 74

4.4 Case Study I — An Interactive Visualization Environment for Finan-

cial Modeling on Heterogeneous Computing Systems . . . . . . . . . 76 4.5 Case Study II — An Interactive Remote Visualization Environment

for an Electromagnetic Scattering Simulation on a High-Performance Computing System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

4.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

5 Integrating World Wide Web and Distributed Computing 80 5.1 Introduction to WWW Client-Server Model . . . . . . . . . . . . . . 81 5.2 WWW and Java: A New Paradigm Shift in Distributed Computing

and Software Engineering . . . . . . . . . . . . . . . . . . . . . . . . 85 5.3 Web and RDBMS Integration . . . . . . . . . . . . . . . . . . . . . . 90 5.4 Case Study I — A Web and Parallel RDBMS-Based Information Sys-

tem for Internet Collaboration . . . . . . . . . . . . . . . . . . . . . . 94 5.5 Case Study II — WWW Search Systems Using SQL*TextRetrieval and

Parallel Server for Unstructured Data . . . . . . . . . . . . . . . . . . 95 5.6 On-going Research and Development in WWW Software Engineering 97

6 A Meta-Computing Software Development Environment 99 6.1 Software Integration Approach in Meta-Computing . . . . . . . . . . 99 6.2 A Layered Meta-Computing Software Engineering Environment … 102 6.3 JavaFlow — A Java Implementation Framework of the Dataflow Model

on WWW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 6.3.1 A JavaFlow Virtual Machine (JavaFlow VM) . . . . . . . . . 105 6.3.2 JavaFlow Servers . . . . . . . . . . . . . . . . . . . . . . . . . 108 6.3.3 JavaFlow Modules . . . . . . . . . . . . . . . . . . . . . . . . 110 6.3.4 A JavaFlow Protocol for Dataflow Message Passing . . . . . . 111 6.3.5 JavaFlow API . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 6.3.6 A JavaFlow Runtime Support Facility . . . . . . . . . . . . . 114 6.3.7 A JavaFlow Module and VM Node Registration Subsystem . . 116 6.3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

7 Conclusion 119

A NPAC Meta Computer 122 A.1 Parallel Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 A.2 Workstation Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 A.3 Desktop Workstations . . . . . . . . . . . . . . . . . . . . . . . . . . 126 A.4 Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 B Case Studies of Multi-Paradigm Programming in the Software Inte-

gration Framework 130 B.1 Parallel Programming on Connection Machine CM5 . . . . . . . . . . 130 B.2 Parallel Virtual Machine (PVM) . . . . . . . . . . . . . . . . . . . . . 134 B.3 Comparison of Numerical Advection Models . . . . . . . . . . . . . . 134 B.4 Option Price Modeling on an IBM SP2 and a DEC Alpha Cluster . . 137 B.5 Distributed Computational Electromagnetics Systems . . . . . . . . . 140

B.5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 B.5.2 The RCS Application Problem . . . . . . . . . . . . . . . . . . 143 B.5.3 Parallel MoM Implementation Strategy . . . . . . . . . . . . . 144 B.5.4 Performance Comparison and Discussion . . . . . . . . . . . . 151 B.5.5 A Scheduling and Visualization Framework for the RCS Simu-

lation on a Meta-Computer . . . . . . . . . . . . . . . . . . . 153

C Case Studies of Distributed Data Visualization and Parallel Processing in the Software Integration Framework 159 C.1 A Financial Modeling Application . . . . . . . . . . . . . . . . . . . . 159

C.1.1 Problem Description . . . . . . . . . . . . . . . . . . . . . . . 159 C.1.2 System Configuration and Integration . . . . . . . . . . . . . . 160 C.1.3 Performance Analysis . . . . . . . . . . . . . . . . . . . . . . . 167

C.2 A Computational Electromagnetic Application . . . . . . . . . . . . . 167 C.2.1 Problem Description . . . . . . . . . . . . . . . . . . . . . . . 167 C.2.2 Parallization of the EMS Simulation Model . . . . . . . . . . . 168 C.2.3 Implementation of the Remote Visualization Environment . . 173 C.2.4 Interstage Communication and Synchronization . . . . . . . . 177 C.2.5 Performance Analysis . . . . . . . . . . . . . . . . . . . . . . . 179

D Case Studies of Integrating World Wide Web and Distributed Computing 181 D.1 Parallel Oracle 7 RDBMS Servers on IBM SP2 . . . . . . . . . . . . . 181

D.1.1 The IBM SP2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 D.1.2 Virtual Shared Disks . . . . . . . . . . . . . . . . . . . . . . . 182 D.1.3 Oracle Unix Distributed Lock Manager (DLM) . . . . . . . . . 184 D.1.4 The Oracle7 Parallel Server (OPS) . . . . . . . . . . . . . . . 185

D.2 A Web and Parallel RDBMS-Based Information System for Internet Collaboration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 D.2.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 D.2.2 System Configuration . . . . . . . . . . . . . . . . . . . . . . . 190 D.2.3 Search Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . 193

D.3 WWW Search Systems Using SQL*TextRetrieval and Parallel Server for Unstructured Data . . . . . . . . . . . . . . . . . . . . . . . . . . 199 D.3.1 Components of a WWW search system . . . . . . . . . . . . . 199 D.3.2 Oracle SQL*TextRetrieval . . . . . . . . . . . . . . . . . . . . 206 D.3.3 A Prototype WWW Search System . . . . . . . . . . . . . . . 208

Bibliography 215

List of Tables

B.1 Processing Phases of Sequential ParaMoM and Their Computational

Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 B.2 Timings of parallel I/O and LU solver for N = 2000 (in seconds) … 150 B.3 Timing Comparisons for N = 4889 (in seconds) . . . . . . . . . . . . 154 B.4 Timing Comparisons for N = 3060 (in seconds) . . . . . . . . . . . . 154 B.5 Timing Comparisons for N = 1504 (in seconds) . . . . . . . . . . . . 154 B.6 Timings of Different N on 2 SUN IPXs Interconnected by an ATM Link155 C.7 Experimental timings for the option price modeling (in seconds) … 167 C.8 Timings of calculations and communications for the EMS (in seconds) 180

List of Figures

1.1 Layered software architecture for a PSE for PDE applications …. 16 2.1 The von Neumann model vs. a dataflow model . . . . . . . . . . . . . 23 2.2 A coarse-grained dataflow model . . . . . . . . . . . . . . . . . . . . . 24 2.3 Sequential dataflow vs. concurrent dataflow . . . . . . . . . . . . . . 30 2.4 Internal structure of a module . . . . . . . . . . . . . . . . . . . . . . 34 3.1 Computation and simulation as a series of mappings . . . . . . . . . . 50 3.2 Integration of multiple programming models as a single mapping … 51 3.3 A general AVS/CM5 integration model for multiparadigm programming 56 3.4 The parallelism hierarchy in the AVS/PVM system . . . . . . . . . . 60 3.5 Serial AVS I/O vs. parallel node-node channels for intermodule com-

munication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 4.1 A general visualization/computing model . . . . . . . . . . . . . . . . 73 4.2 A general performance model . . . . . . . . . . . . . . . . . . . . . . 74 4.3 Different cases of the pipelined calculations and communications … 75 5.1 The WWW Client-Server Architecture . . . . . . . . . . . . . . . . . 82 5.2 The Web client-server model as a special case of the dataflow integra-

tion model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 5.3 Linkage of Oracle RDBMS servers to the Web . . . . . . . . . . . . . 93 6.1 A Three-Layered Meta-Computing Software Development Environment 104 6.2 The JavaFlow Architecture and Major Components . . . . . . . . . . 106 A.1 NPAC meta-computer - processing nodes . . . . . . . . . . . . . . . . 123 A.2 NPAC meta-computer - interconnections (networking) . . . . . . . . . 128 B.1 Components of a CM5 system . . . . . . . . . . . . . . . . . . . . . . 131 B.2 System configuration of the prototype system . . . . . . . . . . . . . 135 B.3 A graphical user interface of the prototype system . . . . . . . . . . . 136 B.4 System configuration of the interactive distributed financial modeling

prototype . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 B.5 The Graphical User Interface . . . . . . . . . . . . . . . . . . . . . . 141 B.6 Program structure of the ParaMom implementation . . . . . . . . . . 147 B.7 Patch-patch interaction and the Z-matrix . . . . . . . . . . . . . . . . 147 B.8 Fill Time v.s Processors (Matrix Size = 988) . . . . . . . . . . . . . . 152 B.9 A Integrated Scheduling, Visualization and Simulation Framework on

a Meta-Computer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 B.10 A graphical output of the prototype system (a VFY) . . . . . . . . . 158 B.11 The graphical user interface of the prototype system (a sphere …. 158 C.1 System Configuration 1 for the Financial Modeling on a CM-5 and a

DECmpp-12000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 C.2 The graphical user interface for the option price modeling (1) …. 163 C.3 System Configuration 2 for the Financial Modeling on an IBM SP2

and a DEC Alpha Farm over NYNET . . . . . . . . . . . . . . . . . . 165 C.4 The graphical user interface for the option price modeling (2) …. 166 C.6 Physical parameters of the EM scattering problem . . . . . . . . . . . 169 C.7 The System configuration for the EMS application . . . . . . . . . . . 174 C.8 A Graphic user interface for the EMS visualization system . . . . . . 176 D.1 Parallel Oracle7 Server Architecture on an IBM SP2 . . . . . . . . . . 183 D.2 The DLM Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 185 D.3 Configuration of the Integrated Internet Collaborative Server . . . . . 191 D.4 Search Interfaces for the USENET Newsgroup and Mailing list Database194 D.5 A Frame-Based Search Interface . . . . . . . . . . . . . . . . . . . . . 195 D.6 A Form-Based Search Interface . . . . . . . . . . . . . . . . . . . . . 196 D.7 A Hypermail-Like Search Interface . . . . . . . . . . . . . . . . . . . 197 D.8 A Calendar-based Search Interface . . . . . . . . . . . . . . . . . . . 198 D.9 Components of a Web Search System . . . . . . . . . . . . . . . . . . 200 D.10 An RDBMS-based Web robot system . . . . . . . . . . . . . . . . . . 209 D.11 The search interface of the NPAC Web search system . . . . . . . . . 212 D.12 Display of search results and an interface of the navigation and refining 213

Chapter 1

Introduction

In this introductory chapter we first outline the fundamental questions in parallel computing that motivate this thesis work and identify the key research issues to be studied. We discuss the role software integration plays in solving real-world applications on high-performance parallel and distributed systems. Viewing software integration as essentially part of the conceptual problem-solving process and environment in mapping meta-problems of complex applications onto meta-computers that consist of disparate parallel and distributed computing elements, we focus our discussion on basic features and software engineering requirements found in the two complex systems.

1.1 Problem Definition

Today’s parallel software systems comprise islands of automation. All software components perform some limited range of useful functions, but components usually do not interoperate effectively. Although the driving force behind the adaption of parallel computers is the performance requirement, most real-world applications demand a sophisticated software system that provides both high performance and versatility. For example, parallel software systems are expected to have graphical user interfaces to incorporate complex data types, to require the use of multiple programming models, paradigms and parallelisms, and to have integrated database capability as well as many other particular functionalities. A general parallel software system is a toolbox that can be customized and optimized for particular application needs. On the other hand, parallel computers are very specialized systems. Different vendors usually develop different software. One of the major problems in parallel software engineering is how to provide software interoperability and system customization beyond the single-platform capability. Software integration, i.e., how to combine or “glue” various pre-existing components into an integrated environment, is the main topic of this work.

As an effective parallel computation model, dataflow has been widely studied in computer architecture and in compiler and programming languages, yet little previous work has been done on using a dataflow model as an integration framework. Our major approach to software integration is built on a dataflow execution model. Some fundamental questions to be studied in this thesis include:

1. Can a coarse-grained dataflow programming paradigm be applied to a software integration model? If the answer is “yes,” what are the major benefits and drawbacks as compared to other approaches?

2. What are the key issues in building a dataflow-based integration system? More specifically, what applications are best suited to the dataflow integration model?

3. Can the dataflow model provide a complete coordination framework with which to integrate coarse-grained task-parallelism and fine-grained data-parallelism?

Parallel processing of an application requires partitioning and allocating the underlying computation for the targeted architecture. Parallel processing provides many alternative solutions to a specific problem. Determining the efficiency of such solutions in a reliable way is a crucial goal. Current methodologies for parallel computing are too complex and too specialized to be accessible to most engineers and scientists. A major areas that requires software integration is a Problem-Solving Environment (PSE), an integrated collection of software tools that facilitates problem-solving in some domain. A PSE must hide most or all of the considerations of load balancing, parallelization of algorithms, distributed memory, message passing, etc. Based on extensive case studies in building several application-specific PSE’s, this thesis intends to answer the following questions about building a general PSE on HPDC systems: Is the proposed dataflow-based software integration framework sufficient for a general PSE? What are the component technology and parallel software integration requirements for a PSE in terms of functionality and performance? Can the integration of Web-based, coarse-grained, distributed computing and fine-grained parallel programming languages provide a complete PSE solution to general HPDC applications?

1.2 Meta-Problems and Meta-Computers

Meta-problems can be defined as complex problems that require significant computational power and space not possible on a conventional uniprocessor computer or single supercomputer within the required time. In other words, meta-problems are a class of applications to be solved on high-performance parallel and distributed computing systems. Grand Challenge problems, for example, which are complex problems of national societal importance that demand more resources than can be provided by any one conventional computer, and National Challenge applications are among the meta-problems. While this definition may be fuzzy in terms of “big” and “complex” characteristics, they share some common features:

1. Complex problem structures and domain In terms of problem structures and domain knowledge, meta-problems are usually nonuniform, irregular, and heterogeneous. In most cases there is little or no structured representation at the topmost level and their components are loosely coupled with each other, although within a subdomain the structure may be well defined. It is usually anticipated that some parts of a meta-problem may be highly sequential, while others are parallel. Most meta-problems are multidisciplinary or interdisciplinary applications in science and engineering. A combination of simulation models and numerical methods may be required to solve the overall problem in which each subdomain component may need a different set of problem-solving techniques with little or no commonality. The interface among individual disciplinary applications can be quite complicated and sometimes nonlinear. A good example of a meta-problem can be found in the Multidisciplinary Analysis and Design (MADIC) project developed at NPAC [40]. In this project the analysis and design of large-scale, complex products such as aircraft and automobiles require a combination of multiple engineering and scientific disciplines that include fluid dynamics, structural dynamics, heat transfer, combustion, and manufacturing. Another example is the NASA Grand Challenge data assimilation application, jointly developed by NASA Goddard, NPAC, and JPL [41] to simulate important phenomena associated with environmental global changes such as greenhouse-gas, ozone loss, and atmospheric pollution. In this application, in each step of a step-wise analysis a Semi-Lagrangian transport model is tightly coupled with an Optimal Interpolation model. The transport model is physically motivated in an icosahedral 3D sphere geometry whose algorithm has significant irregular and dynamic computational requirements. An optimum interpolation analysis technique is used to interpolate data from random spatial locations of approximately 100,000 observations to a regular grid. Yet another example in much smaller scale but representative, to be elaborated on in Chapter 3, is a Moment-Method-based radar cross-sectional (RCS) problem in computational electromagnetics [78]. The problem structure of this application can briefly be described as follows: A finite element method using triangular patches is used to discretize the threedimensional geometry surface of a RCS target in arbitrary shape, and a stencil integral is calculated to give the EM characteristics at each point. The result is represented in a 2D matrix, and a matrix solver in linear algebra is used to solve the linear equations representing the results of the finite element approach in the former part. In this case, two different methods — finite element and linear algebra — are emloyed to formalize the problem structure, and a matrix is used to link the two components linearly. In terms of the way in which problem components interact with each other, meta-problems are usually an asynchronous collection of loosely synchronous components where each component program module may be parallelized in a fine granularity of data and task decompositions.

2. Computational time and space requirements The computational complexity of meta-problems is usually enormous. They tend to require more physical space, finer resolution, more accuracy, and faster response time to be able to realistically model/simulate real world problems or uncover new phenomena in science, tasks that were previously impossible or unrealistic on a single conventional computer or supercomputer. For example, in Chapter 4 we note in our description of a financial modeling application that, despite the fact that they are far more accurate than any model popularly used by the commercial market, option-pricing models based on either the Monte Carlo or the binomial method are used in academia for theoretical purposes due to the enormous CPU requirements of the algorithm. Only since the breakthrough in computational power in high-performance systems has it become possible to bring them into practical applications in industry. Many Grand Challenge applications prove that realistic modeling will be possible only with orders of magnitude increase in computational performance [59].

3. Rich data source and data types A meta-problem may be computationally intensive simply because the volume of data to be processed is enormous. For instance, for many National Challenge applications information processing, rather than simulation, is the the major demand. One example is the so-called “data mining” application where strategic decisions are based on the processing and analysis of a large amount of historical data that require high-performance systems with powerful processors, advanced database management, and intelligent analytic methods. It is expected that by the end of the decade high-speed networks will enable the integration and coalescing of vast amounts of information stored in diverse databases. This will provide unprecedented new opportunities to acquire new knowledge by applying various data-gathering and data-mining processes to such massive databases. Data types include not only text and binary but all sorts of multimedia data such as images, audio, and video. These goals will likely be met, but not without significant advances in our understanding of how to build and integrate high-performance parallel database management systems, parallel and distributed I/O, artificial intelligence, and multimedia applications such as Web-based applications that are efficiently executed on a variety of parallel and distributed hardware architectures. In Chapter 5 we will describe such an application in Web/RDBMS integration, building a Web search/archive system for structured USENET newsgroups, and an unstructured full-text Web document on a parallel database management system.

4. Problem abstraction and task decomposition requirements The creation of higher-level problems and machine abstraction is a fundamental concept in computer science and engineering. An abstraction is a simplified interface that provides access to some underlying lower-level mechanism. It is also a representation of problem entities or components that are relatively independent and logically isolated. There is usually a hierarchy of task and data abstraction associated with each level of problem abstraction in a topdown fashion. Abstractions are useful for managing complexity and for analyzing system architecture. Higher-level abstraction that is closer to problem structure gives a better structural overview and understanding of a complex problem. Lower-level abstraction that is closer to the machine and implementation structure gives better details on exact implementation efficiency and machine dependency. A hierarchy of task and data abstraction is essential to system analysis, software development, and system integration. The refining process of such abstractions is actually the realization of the design-mapping process from an application description to its computer implementation. Parallelism may exist at every level of the abstraction problem. A good mapping scheme/methodology should preserve inherent parallelism in the problem domain while utilizing efficient implementation on parallel systems. Different computer implementation stages may require different levels of task and data abstraction.

5. Multiple functionality and resource requirements Parallel computation is not the only task in a meta-problem. A complete realworld application may need to support diverse functions that require multiple computing resources such as processor power, I/O, networking/communication media, programming models, database management, graphics and data visualization, real-time simulation and modeling, etc. The demand may come from

(a) a variety of computing services/resources to be employed;

(b) interoperativity and interfaces among components; or

(c) functional parallelism.

A parallel software support environment should be able to combine different computing services in an effective and coherent way.

The hardware computing environment provided by vendors usually consists of heterogeneous computing platforms with associated data storage devices linked by communication channels. The computing environment needed by application developers supports rapid porting or prototyping of software codes that can make efficient use of vendor-supplied hardware. The software infrastructure that normally provides the interface between application developers and computer software seriously lags behind hardware advances. Difficulties also exist in providing system software for heterogeneous platforms, since system software has historically been developed by vendors who optimize the software for their own platform. This has become further exacerbated by bottlenecks in the development of application software because of associated paradigm shifts persistently occurring, which include:

1. Development of a range of parallel computers with different architectures and power.

2. Integration of either homogeneous or heterogeneous computing platforms into meta-computers. The meta-computer is a uniform interface to disparate computing platforms, providing the application programmer the opportunity to decompose the application problem across multiple machines.

On the other hand, different problems will generally run most efficiently on different computer architectures. It is generally agreed that there is no single parallel system architecture that can handle universal applications of any type most efficiently. A range of different architectures will be available for some time to come, including vector supercomputers, SIMD massively parallel systems, MIMD distributed-memory machines, shared-memory SMPs, and networks of workstations.

The user would prefer not to have to deal with details of the different hardware, software, languages, and programming models for the various classes of machines. Therefore, the aim of system and software integration is transparent distributed computing, sometimes called “meta-computing,” to provide simple, transparent access to a group of machines of different architectures connected by a high-speed network to each other and the outside world, and to data storage and visualization facilities. Users should be presented with a single-system image so they do not need to deal with different software, languages, software tools, and libraries on every machine. They should also be able to run an application across different machines on the network.

Meta-computing implies not only different computer architectures but different languages, new software, and new libraries. It will open up computation to new fields and new applications, and offers a different way of viewing problems. Virtually all complex real-world meta problems are inherently parallel in that many different elements of the problem domain interact with one another at any given time. In a sequential language the structure of the problem must be artificially broken up to fit within the confines of the sequential computer, for which only one computation can occur at a time. Using parallel computers and parallel languages allows the programmer to better preserve the problem structure in the application, and also in the algorithm.

To solve complex and computationally (or informationally) intensive problems such as meta-problems, we need to employ a whole computing infrastructure and a software environment to bear on those application requirements at very high performance. We therefore define “meta-computer” as a framework of a system infrastructure and a software engineering environment that is composed of a heterogeneous collection of disparate computer systems and data storage systems integrated by highbandwidth LANs, all at one site. Associated with the meta-computer infrastructure is a parallel software engineering framework consisting of a set of resource management and programming support software necessary to effectively apply computing facilities to meta problems. The software integration framework to be discussed in this thesis is one of such efforts to build a meta-problem solving environment on meta-computers.

A meta-computer can be as small as a single parallel system or as big as all of the facilities in a supercomputer center. As an analogy to a uniprocessor system, a meta-computer has the following major computing components:

1. CPUs (distributed and heterogeneous in size and power vs. single CPU)

2. RAMs (distributed memory components vs. integrated chips in VLSI)

3. Interconnections (inter-networking vs. BUS)

4. I/O channels and peripheral data storage devices (loosely distributed versus tightly centralized)

Meta-computers share the following common features:

1. Heterogeneity. The computing elements in a meta-computer may be from different vendors with different architectures and varied capacity for speed, power, bandwidth, configuration, etc. The motivation for migrating to heterogeneous environments is the potential to apply the most suitable computing systems to the different parts of a given program, thus better matching resources to computing needs. The majority of computationally intensive applications perform well on parallel computers by taking advantage of the simple idea of “data parallelism,” which means obtaining concurrency by applying a particular algorithm to different sections of the data set concurrently. Data-parallel applications are scalable to a larger number of processors for a larger amount of data. Another type of parallelism is “functional parallelism,” where different processors (or even different computers) perform different functions or different parts of the algorithm. Here, the speedups obtained are usually more modest and this method is often not scalable; however, it is important, particularly in multidisciplinary applications. Surveys of problems in computational science have shown that the vast majority (90%) of applications can be run effectively on MIMD machines, and about 50% can be run efficiently on SIMD machines. Currently, there are many different parallel architectures, but only one — a distributed memory MIMD multicomputer — is a general, high-performance architecture known to scale from one to many processors [58]. Not only are the computing elements in a meta-computer heterogeneous, the programming environment on which it is supported is expected to be quite different from other environments. The broad range of HPCC applications may embody a rich diversity of program structures, and it is anticipated that no one parallel programming model will satisfy all needs. Data-parallel, task-parallel, and object-parallel forms have all found applicability to end user problems and are expected to continue to do so. A parallel programming environment will incorporate them into a single schema that provides means for a single application to use different languages or different parallelism models for separate parts of a problem. Porting and rewriting applications programs also require a heterogeneous support environment that encourages code reuse, portability among different platforms, and scalability across similar systems of different size. Necessary to achieve this is an integration framework that seamlessly couples key programming elements that include parallel I/O and message-passing primitives as well as parallel programming support tools. This framework will of necessity support interoperability among independently derived program modules running on separate computers in a distributed heterogeneous computing environment.

2. Distributed elements. In a meta-computer almost all elements are distributed. Nothing is shared except the interconnections binding them together into a single system image. The shared-nothing or decentralized feature makes a meta-computer different from a distributed-memory MIMD machine, on which most base-operating system software is still tightly integrated and centralized with the distributed architecture. A software integration system on a metamachine is very similar to the OS software on an MIMD machine, except that it must deal with all the shared-nothing components and create a shared software environment for applications. Still, the degree and level of distribution is in a hierarchy so that all software technologies developed on a particular system can be reused and integrated into this much broader meta-level environment.

3. Coarse-grained. The characteristics of distributed and heterogeneous software and hardware components in a meta-computer determine that they are in coarse grain. Task parallelism and functional parallelism are the only sources of parallelization that can be exploited at the highest level of an integration framework.

An example of a meta-computer can be found in Appendix A where the NPAC in-house computing facility is described. Most of our case studies in the later chapters are built on this meta-computer.

1.3 Software Integration for Meta-Problems and

Meta-Computers

Solving a complex problem on a complex system requires a sophisticated software engineering environment. The computing environment areas include data storage, communications, and programming support environments. Applications are inherently limited by the capabilities of the computing environment and access it provides.

Standardization of the programming environment is complicated by the competing factors of granularity versus locality for both task and data. Task or data granularity is a decomposition metric with the issue of whether one should restrict fine-grained decompositions to local environments (i.e., within one machine) or to loosely coupled machines. Task or data locality is a task/data distribution metric that addresses the issue of whether a particular application will work well on distributed memory architectures. Since application algorithms may require a particular task/data locality or granularity pattern for efficient execution, there may be multiple programming environments that will need to be supported.

Programming environments for meta-computers must go beyond conventional parallel languages and message-passing programming in expressivity and program execution. In addition to the normal requirements, programming environments must incorporate semantic constructs for delineating program parallelism, identifying locality and specifying data set partitioning, coordinating disparate components, and so on.

The approach we propose, to be elaborated on in later chapters, is a dataflow-based software integration framework that stems from other examples arising in graphics dataflow systems such as AVS/Khoros. We believe that successful software integration models should be built around meta-problems and not around meta-computer architectures. Parallel languages and tools should be able to map problem components containing various diverse structures into a range of computer architectures, transparent to the software integration framework and to programmers.

We define software integration as the sum total of the assimilation effort to plan, procure, and construct a fully operational computing environment by combining preexisting and new program components for a complex application into a single unified programming framework in a seamless way so that the application is accomplished effectively. We can view parallel software integration as a framework with which components of a meta-problem are mapped onto computing elements of an metacomputer in a single integrated environment so that various distributed computing resources are optimized to solve the application problem.

Software development on HPDC systems has in the past been focused on developing efficient parallel algorithms and specialized software components to run fast on a parallel system. A stand-alone program comprising a command-line interface and a few specialized functions was usually sufficient for a successful parallel software application. However, real-world applications such as meta-problems demand more than just parallel algorithms. It is expected that parallel software systems will have graphical user interfaces; software systems will incorporate complex data types such as imagery, graphics, and other multimedia; and a parallel software will require integrated database capabilities to manage and process a large data volume. Heterogeneous computing environments are a reality in most organizations where differences between platforms reinforce the need for multi-vendor integration solutions. Most systems solutions today require a great many components, often from a variety of product vendors operating in harmony with one another. The process of software integration is the optimization of disparate software and components toward the construction of a seamless operation to ensure that the components of the computing environment are well-matched, complementary, and tuned for optimal performance.

Research scenarios in system and software integration include the following:

1. Distributing portions of a given job between platforms so that each portion runs on the most amenable platform: for example, using a CRAY C90 and a Paragon to solve a quantum chemistry reaction dynamics calculation over a Gigabit testbed.

2. Solving a problem on one platform and transferring the data to a second platform for processing, visualization, and archiving; for example, to conduct an ocean modeling with the problem solved on the CRAY T3D and the output data visualized on a different machine.

3. Obtaining remote-sensing data from an archive on one platform, then processing the data on a second platform.

The scale of these meta-problems requires integrating a variety of needed resources into a single, easy-to-use capability that spans parallel and distributed computation, data access, data manipulation, and data visualization. This integration can result in decreased computation time, decreased data-processing time, decreased time to interact with the data to massage it into a usable form, and decreased time to interpret the results. In short, this integration allows the researcher to solve the problem much more quickly, resulting in a more rapid prototyping process.

Such capabilities can be constructed by linking resources at the application level through a heterogeneous computing environment software layer or at the operating system level by explicit modifications to each of the linked platforms. The former is expected to provide an easier migration path for new hardware and software systems. The latter is expected to provide higher performance. Either approach requires the development of a significant amount of software infrastructure, and both require the solution of similar underlying problems. We adopted the former approach in our work. In Chapter 2, Section 4, we give a brief review of similar work in software integration approaches adopted by other research groups.

1.4 Software Integration in a Problem-Solving En-

vironment

One can view software integration from the perspective of building a problem-solving environment on HPDC systems.

Research in problem-solving environments (PSE) has been active since the 1960’s, soon after high-level programming languages were invented [46]. In general, a PSE is a computer system that provides all the computational facilities necessary to solve a target class of problems. Major features in a PSE include advanced solution methods, automatic or semiautomatic selection of solution methods, ways to easily incorporate novel solution methods, transparent use of underlying computer software and hardware, and a framework to solve simple or complex problems and for rapid prototyping or detail analysis.

While most of the research and practice of PSEs in the 1970’s and 1980’s was hindered by the lack of computing power, advances in high-performance computing, combined with a better understanding of computing and computational science in the 1990’s, have put PSEs well within our reach. The recent refocus of the US HPCC program on national challenges has generated new incentives in this area in which PSEs can help build large-scale application systems in areas where problems and their solution methods are well understood and efficient (parallel) computer algorithms have been found and widely practiced.

One of the most important issues in a PSE is the software architecture used to assemble, organize, and orchestrate system components to allow gradual growth and evolution during its life-cycle. Building a PSE on a distributed parallel computing system such as the meta-computer tends to be more difficult and challenging than on a uniprocessor computer, as it now has to deal with the system complexity at both ends.

A software integration in a PSE will need to support the following key capabilities:

1. Rapid prototyping for architecture specialization and customization: a PSE CHAPTER 1. INTRODUCTION 16

Figure 1.1: Layered software architecture for a PSE for PDE applications should allow a user to rapidly integrate and connect modules consisting of computation kernels, libraries, and user interfaces.

2. Portable interfaces to avoid monolithic code: PSE modules should be able to execute on networked machines. A constructed PSE can consist of modules running on a variety of modules simultaneously by using the computing and visualization resources most suitable to each. A PSE can also contain multiple copies of modules running in parallel, as well as modules that contain parallel code.

3. Efficient module resolution to allow a user to create customized modules, to connect modules running on remote machines, and to select modules for different purposes.

4. Capture the complexity of systems while hiding complexity and heterogenity: an integration frame is a layered collection of program objects that work together to hide increasing levels of complexity within target problem domains.

A software component architecture plays a critical role in building a PSE. For example, Figure 1.1 gives a three-layer software architecture for the development of PSEs for PDE-based applications [45]. At the top layer, a set of tools is provided to specify the input and to interact and observe various facets of the “hybrid” numerical and experimental models used to simulate various applications. These tools allow users to solve problems by communicating in the user’s terms. The bottom layer involves the various meta-libraries for the numerical simulation of various physical objects and knowledge bases that support the computational intelligence of domainspecific workbenches. These libraries are portable on specialized, generic, and virtual machines. Between the two layers sits a middle-ware whose objective is to glue a range of independent subsystems and tools. A software integration framework is required to lay down the backbone for this software architecture, and it is this middle-ware that plays the integration role in a PSE environment.

The building blocks of a PSE are of various component technology, general or specific for application domains. Examples of generic components technology include:

1. User interfaces, such as World Wide Web, X-window, Motif

2. Database or knowledge-based systems such as Oracle, Illustra, expert systems

3. Programming languages and compilers such as Fortran90, PVM, MPI

4. Numerical libraries such as LINPACK, ScaLAPACK, CMSSLL

5. Collaborative tools such as IRC chat, whiteboard

Different PSEs may need to combine different component technologies. However, a universal PSE architecture must allow “plug-and-play” of new algorithms or data structures so that it does not exclude key technologies. In addition, a PSE framework should support a distributed software development approach with synergistic contributions from loosely coupled groups of researchers. Research in software integration will rely on advances in the following computer science areas:

1. Object-oriented technology such as CORBA, JAVA, C++

2. Visual programming and GUI such as Visual Basic, Delphia, AVS

3. Software engineering methodology

4. Networking and distributed computing

Support for interoperability is critical in PSE. By interoperability we mean the ability of two or more programs to communicate or work together despite having been written in different programming languages or models. The need for interoperability arises in many contexts of a PSE. Generally, the desire to combine programs written in different languages springs from the availability of specific capability in some particular language, programming model, processor, or existing program. For example, the number-crunching power of a vector processor and the availability of a particular numerical analysis package in Fortran90 might entice a programmer to attempt interoperation of a PVM program running on a cluster of workstations and a CMFortran (CMSSL) code running on a CM5 with vector-processing capability. One of the common goals in a PSE and software integration is to seamlessly facilitate software and application interoperability.

A General Software Integration

Framework

As we discussed in the previous chapter, a general semantic model of program execution is needed that can serve as a framework and programming paradigm for the integration of program components written in high-level parallel languages or in portable message-passing systems on a meta-computer. In this chapter we first illustrate how in dataflow concepts functional programming ideas may be directed to exploit the power of parallel and distributed systems and have the potential to lead to an important universal semantic model of computation and high-level, coarse-grained software execution model. We then propose a general software integration framework based on the dataflow model. We describe in such a framework the major features needed, the advantages a user can gain, and the key issues and requirements to be addressed in building the framework. At the end of the chapter we review similar work in a parallel software integration framework and give a detailed description of the Application Visualization System, a framework testbed to be used in later chapters to develop various application systems and to apply concepts in the proposed integration model to certain more specific software integration contexts.

CHAPTER 2. A GENERAL SOFTWARE INTEGRATION FRAMEWORK 21

2.1 Introduction to Dataflow Programming

Studies in dataflow programming have been active ever since the limitations of the conventional von Neuman architecture were found in the 1970’s [79, 42]. As an alternative to and departure from the control-driven programming model inherent in the von Neuman architecture, dataflow models have been extensively studied and practiced in various fundamental areas of computer science, including computation models, programming languages, machine architecture, compilers, and parallelism. Many early researchers in this area believed that the dataflow model would become a new and general model of computation, be able to exploit all aspects of parallelism arising in general purpose programs, and that it would enable the development of highlevel languages in which the programmer need not manage every detail of efficient mapping of programs and data on a parallel machine [43]. Most research and papers have been devoted to the architecture and language aspects of a dataflow model in fine-grained parallelism and program execution. Indeed, the dataflow concept has been found in many applications in conventional computing such as the pipelining and multiple instruction issue techniques used in many RISC processors. Dataflow programming was once a promising approach to a new generation of high-performance computing. The reality is that it is still not mature enough to be a main-stream technology in general parallel computing, and it is not widely accepted and practiced in the HPDC community and industry. It is anticipated that von Neuman-based processors, on which current parallel technology is built, will continue to dominate high-performance computing and applications for many years to come.

However, the dataflow concept has many advantages and unique characteristics that can be borrowed to use in today’s parallel software environment. We believe that a hybrid parallel software system, combined with a conventional control-driven model and dataflow programming, will likely complement each other and be able to provide better software and hardware solutions to parallel software for general application problems that require high-performance computing. One of the major findings in this thesis shows that a dataflow model can fit naturally into an HPDC software engineering environment as a unified software integration framework. In this chapter we focus our discussion on adopting a dataflow model as a programming paradigm and execution model.

2.1.1 Dataflow Model As a General Programming Paradigm

Writing parallel programs is currently much more difficult than writing sequential programs. The difficulty comes mostly from three aspects:

1. Managing the control flow along multiple threads of execution

2. Managing the placement and access of global data

3. Synchronizing events in such a way that program execution is deterministic and parallel speedup is achieved

While it is possible for a programmer or end-user to choreograph explicitly a parallel program by indicating which tasks execute when, how tasks communicate and synchronize, and where data should be placed and how it can be accessed, such an undertaking is extremely complex.

Dataflow and data-driven are radical departures from the traditional approach based on the von Neumann or a control-driven computation model. In the von Neumann model, a piece of processor sequentially fetches instructions and data from memory for execution. In the dataflow execution model, a program is represented as a graph in which nodes can be viewed as virtual processors. They execute as soon as the input operands become available on the input edges. There is no concept of memory. The concept of memory in a control-driven model introduces states in which every update of a memory location implies a state transition. This state-driven nature of the von Neumann model makes it a sequential model. In the data-driven model, a piece of data is not represented as a static entity (memory) whose value can be updated time after time. Instead, it is a dynamic entity that is produced once and consumed by another instruction (fine-grained dataflow) or by a set of instructions, i.e., by a program entity (coarse-grained dataflow). This is illustrated in Figure 2.1.1.

Figure 2.1: The von Neumann model vs. a dataflow model

In the data-driven model shown in Figure 2.1.1, a program is represented as a directed acyclic graph in which nodes represent instructions and edges represent the data dependency relationship between the connected nodes. A data element produced by a node flows on the outgoing edge(s) to the connected node(s) for consumption. Node executability is determined by the firing rule, which says that a node can be scheduled for execution if and only if its input data become valid for consumption. In this computing abstraction, every node is purely functional in that it causes no sideeffects. Also, computation is inherently parallel and asynchronous, since any number of nodes can be executed as long as their input data become valid.

A dataflow model can be applied to a fine-grained instruction level in which the basic execution unit is a machine instruction. A fine-grained dataflow requires the support of a programming language and machine architecture for problem representation, efficient program mapping, implementation and execution. On the other hand, a dataflow model can also be used as a unique and simple programming concept at the coarse-grained level in which the basic execution unit is a program module; the message-passing among distributed program modules is restricted in a data-driven semantics. In the coarse-grained dataflow execution model, a distributed application

Figure 2.2: A coarse-grained dataflow model

consists of a graph of modules. Each module may run on a different or the same physical processor and the message passing between any two modules is through the edges. The execution of any module in the graph depends solely on the availability of the input operands. While different dataflow systems require different support in hardware and software environments, we are most interested in adopting dataflow as a general coarse-grained programming paradigm and methodology for the purpose of software and system integration on HPDC systems.

The major advantage of a dataflow model as a general parallel programming paradigm lies in the following features:

1. Data dependencies among programs are represented as data channels that connect programs. Message passing between programs is only via data channels. Each program consists of two distinct and relatively separated components: computation and communication. They can be encapsulated and object-oriented for both computation and communication parts. A dataflowbased parallel program is close to a sequential program. Simplicity is the key feature in dataflow programming.

2. The execution of a program is activated only by the availability of the data object it depends on. Thus the overall computation is inherently parallel and asynchronous.

3. Mapping of a program to a processor is determined solely at runtime by system software. It is also only at runtime that a channel between programs is established to represent the logical relationships among the programs of an application. Thus the overall programming can be much closer to the problem representation and frees the programmer from knowing the underlying machine architecture and implementation details, while it is the system software’s job to choreograph and optimize tasks that are closer to the machine implementation, such as program-to-processor mapping, task scheduling, and coordination.

4. Because there is almost no dependency of one program on another, except for the data objects and data types that flow in and out of a program, software modularity and reusability is highly preserved in the system.

Explicit message passing is the major way in which most current parallel programming systems produce efficient interprogram communications, synchronization, task scheduling, and representation of data dependency among parallel programs. We can view dataflow programming as a restricted message-passing system. In a dataflow model, message passing between two programs is via the flow channel connecting the output ports of an upstream program (A) to the input ports of a downstream program (B). Those ports are predefined so that only data of the same type can be passed. To program A, the message passing of one instance of a data object out of the program is asynchronized so that the computation part of program A can be resumed soon after it puts data on its output ports. To program B, the communication can be viewed as blocked, as it must wait on its input ports for incoming data before it can start a new computation. The dataflow model that requires message passing flows only through the channels based on the firing rule.

2.1.2 Dataflow As the Overall Program Execution Model

The message-passing-based integration model is currently the most effective means for the execution of distributed modules on a parallel system, especially if the modules run on a heterogeneous distributed computing system and are written in different languages. Apart from the synchronization requirement in distributed program execution, functional requirements of a distributed application also make it inevitable that messages will be transfered among different functional components that usually are physically located in different places. For example, it is a common practice to use one machine to collect runtime input data, another to process computation, and yet a third one to display results. At the application system specification level, work-flow together with dataflow has been widely used in system analysis to derive functional system definition and to structurally model the conceptual representation of problem components and their relationships [44]. As a restricted message-passing model, the dataflow model can be used to define the synchronization semantics of a messagepassing system so that the modules’ execution is data-driven with communications of two modules strictly through the dataflow channels with no other means allowed to pass data between modules.

As shown in Figure 2.2, when a coarse-grained dataflow model is applied to a distributed application where most of the program modules are located on different physical processors loosely interconnected by a local network, it lends itself to a naturally suitable program execution model with a restricted message-passing mechanism for general-purpose distributed and parallel applications. This is due to the following features found in a typical HPDC application:

1. Task-parallel programming in coarse-grained parallelism at topmost level. When we are dealing with a distributed computing system with the scale of a meta-computer where all computing units are loosely connected by one or more local networks with moderate bandwidth, the unit ratio of computation to communication performance at the highest level of the system is expected to be high. Message passing is through the local network, instead of the interprocess communication or shared-memory implementation when all modules are located on the same machine. Thus the tasks, like the automous processing nodes, have to be coarse-grained and relatively independent in terms of the frequency of message passing among each other. There will a clear programming and execution hierarchy of granularity, abstraction/expressiveness, and efficiency in the system.

2. Functional programming requirements in task decomposition of an application. Most HPDC applications require a computing resource that may have to be physically located over different hardware/software components, and usually they are event-driven, such as window manager, I/O, or GUI. A dataflow execution best matches the event-driven models in a distributed environment in order to optimize overall resource utilization and performance.

3. Modules for an HPDC application likely written in different languages or programming models. Using a dataflow model to facilitate the interfaces among modules provides a uniform, simple, and portable high-level interface to “hook up” diverse program interfaces that support interoperability among various programming tools.

2.2 A General Integration Framework Based on a

Dataflow Model

Based on our analysis and observation of the generic requirements posed by a metaproblem and the system and software characteristics of a meta-computer in Chapter 1, together with the nice features found in a dataflow execution model, we propose a simple and effective dataflow-based software integration model as the overall framework for meta-computing. We outline the common underlying abstractions and concept for “a module interconnection framework” that uses a dataflow execution mechanism to interconnect distributed coarse-grained parallel software components.

Intuitively, a software integration framework can be viewed as a software bus that presents a standard interface into which distributed modules may be “plugged.” In the same way that a hardware bus presents a standard for electrical characteristics and signal protocols — so that boards with that standard may be plugged together — a software integration framework interconnects software components whose internal properties may remain private as long as its interface matches the “bus” standard.

More specifically, a software integration model is a communication facility among separate distributed modules for a HPDC application. The abstract model is a specification of the interface provided by this facility, and the model implementation shows how those specifications are to be realized for a particular set of programming languages and models, host platforms, and communication media. In this way, decisions concerning the interfacing of modules may be encapsulated separately from the modules, rather than distributing those decisions among the application modules themselves.

We describe an abstracted integration model in this section, leaving the model implementation to the rest of the thesis. This integration framework will be further demonstrated under a different system and application context and through extensive case studies and prototypes in the later chapters.

The design goal of an integration architecture includes:

1. Ease of use. It should support such tools as a module plug-in, template, and a visual programming interface or network editor to compose the flow network visually by “drag-plug.”

2. Software modularity. Software reuse is one of the major features required in a software support environment. Another important feature is system extensibility, which should provide for fully automatic parallelization of trivially/embarrassingly parallel algorithms (with no dependencies).

3. Portability and scalability. This will provide portable API and a paradigm for developing distributed application applications in this integration model on a meta-computer to enable users to make efficient use of distributed resources.

The integration model has the following key components:

1. An execution architecture. An execution model must be defined to specify how distributed modules communicate with one another through synchronization. We propose a coarse-grained dataflow model in which each module is represented as a node in the flow graph, which is in turn an abstraction of the flow execution and process control of an application. Dataflow architecture is composed of modules, ports, channels (connections of ports) as well as an execution manager that is the kernel program to ensure proper execution in a dataflow paradigm. Data is communicated between modules through ports. Each output port has a specific data type, and each input port has a collection of data types it can accept. When connecting an output port of one module to an input of another (usually via a visual programming interface), the user interface queries the target module to discover whether it accepts the type produced. If not, an error is signaled. Output from a single port can be sent to multiple destinations. Each destination is notified when a change occurs and can request data as needed. In some cases, multiple output ports can also be connected to a single port. All of the modules connected to this port are treated identically. Data is cached, or buffered, for all modules on the output ports to reduce unnecessary recomputation, and it can be stored with some reference count to avoid unnecessarily copying. Because the targeted parallel hardware infrastracture is a meta-computer, the number of dataflow piplelines in this software framework, unlike in a single machine environment, is more than one and the overall flow execution is in a concurrent fashion so that task parallelism is achieved at the topmost framework level (Figure 2.3). The dataflow between modules in this framework is demand-driven in that data computation occurs only as modules downstream request recomputation of the data. These recomputations typically are triggered by external events such as new source data arriving or interactive parameter changes that require a new CHAPTER 2. A GENERAL SOFTWARE INTEGRATION FRAMEWORK 30

Figure 2.3: Sequential dataflow vs. concurrent dataflow output to be generated. The change is first reported to the module whose parameter is affected and subsequently reported downstream until it reaches a terminal module. Because of this notification, the terminal module indicates to the system that its outputs are out of date and at some time in the future it is called to update itself. The recomputation can also be requested by the readiness of an output device, in which case the terminal module generates new requests for data as often as the device is ready to accept new data. Because the overall framework is on a distributed meta-computer, to support parallelism and distributed functionality a module that needs new inputs requests them from all its upstream connections and then waits for them to finish their computation. The effect is a ripple of requests back through the network from the terminators and then a ripple of computation forward toward the terminators. When all of the data objects for a module are ready, the user-supplied computation procedure is called. The execution of upstream modules and collection of the resulting data (on the input ports) is invisible to the module writer. All the module sees is that its computation procedure is called with up-to-date input. The degree of concurrency in the framework depends on the fan-out and fan-in to each module in the network. If a module has three inputs, each of them can run concurrently. If each of those three modules in turn has two inputs, all six of those modules can run concurrently. The flow executive (manager) is the central component of the system that determines when to run various modules. Its implementation can be like a distributed operating system scheduler. It basically needs to maintain two resources: a queue of modules that need execution, and a pool of tasks, or threads of control, that can be assigned to execute on one of the modules. The implementation of our architecture is largely modeled by the AVS datadriven model [30], to be discussed further in this chapter. Indeed, we will use AVS as the test bed for most of our case studies conducted in this work. We believe, however, that the framework proposed here has a more general meaning than the original AVS model when it is applied to a broader meta-computer context and HPDC application domain.

2. Modules. Modules are the software components of the distributed system and the building blocks that can be interconnected to build a parallel application. They are coarse grained and at a higher level of abstraction than the fine-grained procedures/functions, but at a lower level than the complete application. The abstraction of applications, which consists of smaller components that can be customized and reused, has become common practice in software engineering and is a theme of object-oriented programming technology. The notion of modules is extremely important to a distributed environment where they encapsulate functional components that are independent of platforms and locations. Program behavior is the focal point of the integration framework in such an abstract way that modules are “black-boxes” that consume input data to produce output data. They can be interconnected to form computation networks for particular tasks where the distributed modules can execute in parallel. In summary, modules receive typed data as inputs and produce typed data as outputs. The only way the other modules can observe or affect a module is through input or output ports/data defined for the module. The rest of the details of the module are hidden. Thus this framework presents to an application builder an abstract interface via module input/output ports. In addition to input and output ports, modules can also have parameters used to control the module’s computation. The parameters attached to a control panel can be accessed via a GUI or any other input peripherals. The parameters used primarily for modules require an interactive graphical input source other than that from the normal dataflow input ports. Modules can be characterized by their input and output connections as follows:

(a) Source Modules. They have no inputs and one or more outputs. They represent such data sources as database access programs or simulations that are producing data directly within the system. Another example is a simulator program that continously generates random data. A source module can be a distributed program to take run-time input from a GUI, a file, an RDBMS, or any other source that feeds a data stream into the application. In most cases, specially customized programming efforts are required to gather input from different sources. The behavior of source modules is mostly event- or data-driven in nature and, except where parallel I/O is used, they are serialized by the source supplier.

(b) Computation Modules. They accept one or multiple inputs from source or other computation modules and perform certain computations that generate one or more outputs. A computation module can be any module that is relatively independent of other modules in the application. Intermodule communication occurs only at the beginning and end of the module computation.

(c) Terminal Modules. They have one or more input connections and no outputs. They represent the outputs of the application, such as graphic display, file, or database write output.

A number of tools should be provided to construct the modules’ “glue” automatically, including a visual programming interface that can be used by developers to configure application flow networks from module executables that are precompiled and stocked. In particular, each module can be described, starting with a template file and a name for the module, its input and data types, and the parameters that can be accessed from the control panel. This approach requires module writers to implement only the functionality required by their algorithm in the module. Inputs and outputs can be represented by arguments based on the description in the template. The module writer or integrator will usually write/integrate the computation procedure/subroutine that performs the fine-grained parallel computations unique to the module. Figure 2 gives an internal conceptual structure of a module.

CHAPTER 2. A GENERAL SOFTWARE INTEGRATION FRAMEWORK 34 Figure 2.4: Internal structure of a module 3. Application programming interface (API). Based on the integration archi-

tecture, an API is needed to allow developers to specify the interface (communication) part of an module that can be plugged into the framework. Wrappers and templates for different languages are the common means used to migrate existing programs to the framework. API is usually provided in the forms of libraries, micros and pre-processing tools. Because the dataflow model is a very restricted message-passing system, its simplicity makes it possible to develop automatic code generators that can be used by developers to wrap existing modules and to generate the interface code segment for new modules.

4. A message-passing protocol in dataflow. The protocol defines when and how data objects are transferred reliably between two dataflow modules that can be either on either the same or different machines. It should guarantee a synchronized data transfer and program execution in a deterministic dataflow computation semantics. While it is implemented as part of the execution runtime system using TCP/IP as the base data transport protocol, there can be many approaches in implementing this message-passing protocol. In general, message passing on the same host should use the most efficient mechanism possible, such as shared memory or message queues. Reliability and performance are the major considerations in the actual implmentation over different configurations of distributed systems. The base protocol should be message-based, rather than stream-based, so that the most recent input messages received for a module can be buffered by the runtime system.

5. Data objects. Data entities flow over pipelines connecting different modules on distributed systems. In the dataflow framework, data objects are strongly typed and usually predefined. The system should define a set of basic data types, and a mechanism should be provided to allow user-defined data types by composing new data types from the basic and existing ones. The data objects will also be used/consumed within a module. Because message passing is restricted to dataflow channels, the data objects, types, and operations can be well defined and restricted to enforce strict type-checking and operation constraints. In graphics applications like AVS, such data are merely predefined data structures or data types by C micros or a set of API libraries. They are designed for particular filter or transform operations of data visualization/graphics, as most system and user-developed modules deal with graphics-related data such as field, geometry, or number. In a relational database management system such as Oracle7, tables/relations are the basic data objects in the dataflow execution model. Operations on relational tables are thus well-defined relational operators such as select, join, project, and union. In the above two examples, the data object is only a data structure or nonobject-oriented data, as not all the operations (methods) defined to operate on those objects are attached to (encapsulated in) the data structure definition itself, but are provided in the form of separate constructs in the programming environment, no matter how atomic they may be. Different applications may require different types of data objects. For a general class of applications, a truly object-oriented data structure may be the best candidate in this framework to ensure that the dataflow programming is really an “object-flow” integration framework. For example, Java [105] can be well used as an API to define object-oriented data that are most suitable to WWW applications.

6. A dataflow network runtime support facility. A visual programming interface contains the kernel of the runtime facility. It is the GUI for creating, revising, and starting a dataflow network that represents a distributed application. One of the major advantages in coarse-grained dataflow programming is the abstraction of “port,” “channel,” and “module.” It provides programming constructs that allow a stand-alone program to be wrapped up and encapsulated, so that its data dependency can be visually represented and configured through a “drag-plug” graphical interface. A constructed flow network via this interface should be captured within a text or script language. This language is used to drive the deployment and runtime execution of the modules on a particular meta-machine configuration. The mapping from a flow network onto a physical distributed system should be done at runtime so that certain scheduling optimization and fault-tolerant execution can be enforced.

7. A module publication/registration mechanism. A module repository should be established to allow module developers to publish, contribute, and share standard and user-defined dataflow modules. Module reuse is an important means for improving the productivity of distributed software development. The dataflow module provides a uniform standard interface to facilitate portable software authoring.

Traditional data-parallel and message-passing interfaces have been shown to give a high performance, but the lack of existing third party software for MPP machines and workstation clusters demonstrates the insufficiency of these interfaces in ease of use.

Several distinct styles of parallelisms can be seamlessly integrated and are important to address in the framework:

1. Data parallel. In the data-parallel programming style, the data set is decomposed into chunks that are distributed among several nodes. In this framework each node processes its subset of the data essentially independently, perhaps passing messages among nodes to synchronize computations or exchange data. Note that this represents something of a departure from traditional usage in that a data-parallel model in the framework may still use message passing, and it does not require a data-parallel compiler as HPF does.

This framework is ideally suited to this programming model, since the data sets are usually in a known, regular, self-describing format. This simplifies the task of decomposing the data set, managing the computations, and locating off-node parts of the data.

2. Pipeline parallel. There can be two types of pipeline parallel application in the framework: (1) time-step granular where, for example, a time step of a simulation is processed by a single module and then sent to the next module, potentially freeing the first module to accept the next time step, and (2) chunk granular where the data set is read in or otherwise produced in pieces (either fractions of time-steps or unrelated to time-steps boundaries), and those pieces sent down the pipe. The chunk-granular method is more useful when the data set sizes for each time step are large compared to the memory of the machine(s). Of course, there are obvious restrictions in the type of processing that can take place in a chunk-granular pipeline parallel application; the processing cannot depend on data yet to be seen, for instance. However, it is always possible to use a demand-driven or pull (as opposed to push) architecture to overcome some of these limitations. In this configuration the last module demands a certain portion of the data set from the previous module(s) until some module is able to satisfy it from data it has locally, or which it is able to compute. We can use the dataflow dependence graph to implement this kind of parallelism by using the same kind of data set decomposition as in the data-parallel style.

3. Explicit message passing. The message-passing approach is simpler, more standard, and will generally give a better performance by giving more control to the user. In this framework this can largely be subsumed by the expanded data parallel model. If the user desires, all data transfer can be done via message passing, eliminating the automatic data decomposition. This may be appropriate for some applications, but in most cases allowing the system to distribute the data will save effort and time.

4. Shared memory. The fourth type of parallelism is the shared-memory model. In this model, data is logically stored in one address space with one or more task runs on each node sharing the data. Data may be physically shared, as in an SMP machine, or virtually shared, in which case page faults on a given node result in a page transfer from another node. Several supports for shared memory can be provided under this framework such as spawn parallel tasks/process instances, manage synchronization, and support message-passing library; the user is free to use whatever shared-memory system is supported on specific hardware. This framework makes no attempt to unify this model with the message-passing model beyond presenting a uniform task and data decomposition model, which will limit the portability of modules written explicitly for a shared-memory model only to shared-memory machines. Since message passing on a shared-memory machine simply involves pointer copying, which is relatively fast, a message-passing model is recommended even for shared-memory machines.

2.3 Similar Work in Software Integration on HPDC

Systems

In this section, in order to illustrate alternatives that can be used in building a software integration framework, we compare the approaches of three primary systems in the context of parallel programming and software engineering environments and integration capability. They are:

1. AVS

2. PVM

3. Fortran-M

AVS [1] is a widely available commercial visualization environment based on a dataflow model for scientific data visualization and process control. It incorporates visualization, graphics, visual programming, process management, and transparent networking into a single comprehensive visualization software and development environment. The AVS dataflow is a model of parallel computation in which a flow network of autonomous processes compute by passing data along arcs that interconnect them by input/output ports of the process modules. Each process module fires autonomously as soon as all the input data it requires arrive on its output ports, thus triggering further computation. Both AVS process control and data transfer are in a modular fashion and are completely transparent to the programmer. AVS provides a data-channel abstraction that transparently handles module connectivity and ports type-checking. The module programmer needs only to define input and output ports in predefined AVS data types by using a set of AVS library routines and micros. Message passing occurs at a high level of data abstraction and only through the input and output ports. An AVS kernel (manager) executes the flow network and coordinates the actual data transfer, which are actually carried out by an underlying IPC or RPC mechanism. When the flow network is executed on a uniprocessor machine, IPC (Inter-Process Communication) protocol is employed. When it is executed under a distributed environment, such as a cluster of workstations in which each process or group of processes may execute on a different machine, RPC (Remote Procedure Call) and TCP/IP sockets are used for message passing in the network. Among other systems similar to AVS are Explorer from SGI [31], Data Explorer from IBM [125], and Khoros [126].

PVM [73] is a message-passing programming system for the development and execution of large concurrent or parallel applications that consist of many interacting but relatively independent components. It can operate on a collection of heterogeneous computing elements (nodes) interconnected by one or more networks. PVM provides a straightforward and general interface that permits the description of various types of algorithms and their interactions, while the underlying infrastructure permits the execution of an application on a virtual computing environment that supports multiple parallel computation models. PVM contains facilities for a concurrent, sequential, and conditional execution of application components. Programs in PVM are portable to a variety of architectures. PVM provides a set of message-passing primitives that can be incorporated into existing procedural languages. Primitives exist for invoking processes, message transmission and reception, broadcast, synchronizations, etc.

Other systems include MPI [13] and Express.

Fortran-M [14] is a small set of extensions to Fortran77 that supports a modular approach to the execution of sequential and parallel programs. It allows the specification of dynamic processes and communication structures for the integration of task and data parallelisms in an application. In a programming language context, Fortran-M provides constructs for creating tasks and message-passing channels, for sending and receiving messages among processes, and for mapping tasks and data to processors. Program processes communicate via established channels. Channels and processes can be created dynamically during the execution. Support of concurrent program modularity that is enforced at the language level and task parallelism are the major motivation and features in Fortran-M.

Similar systems are CC++ [2] and CSP [124].

Besides specific capability in their own strengths, namely, AVS for data visualization, PVM for portable message-passing program development, and Fortran-M for developing modular task parallel programs, all three systems can be used to integrate different services and parallel software components into a meta-problem. However, the capability and approach each system has adopted for this purpose differ significantly in the aspects of system configuration, integration architecture, abstraction of message passing, and the level of integration abstraction.

1. Processor and Task Configuration. Fortan-M has language constructs (semantics) with which a program defines when and how a virtual processor array and a process (i.e., task) array are mapped to each other at runtime. The user program must explicitly declare the mapping. A compiler runtime takes care of the allocation of physical processors. The parallel program model in Fortran-M is MPMD (Multiple Program Multiple Data), while in PVM the default model is SPMD (Single Program Multiple Data), in which the allocation and mapping of processes and processors are implicitly carried out by PVM daemons. A user’s program does not need to handle them explicitly. Instead of built-in language constructs and semantics, as in Fortran-M, PVM provides a set of process management library routines to allow new tasks and processors to be dynamically added/removed from the runtime virtual machine configuration. The module and machine allocation and binding in AVS are configured statically through AVS’s click-and-point visual programming interface called Network Editor. It must be done before the execution of the flow network, as the network configuration is not allowed to change during the execution, although a configuration of the flow network can be saved in a text file for future reuse and modification.

2. Integration Architecture. AVS uses a centralized and process-based model in which modules in separate memory address space are loosely coupled by a central kernel manager. This manager controls and coordinates all the data transfer and overall program execution. PVM uses a distributed and loosely coupled process-based execution model in which tasks on each machine have a unique daemon manager that supervises message passing and interactions with other daemon managers. Fortran-M ties the integration model into the language runtime. It is a tightly coupled non-process-based system.

3. Abstraction of Message-passing. AVS and Fortran-M use a highly abstracted “port” and “channel” concept to represent/define how communication is established and data transfered in the program. A channel is a connection between an input port of a module and an output port of another module such that the only way to communicate among modules is via established channels. A port is a predefined data object associated with a module with certain data types. It is strongly typed so that only data with the same type as the port can be attached. There are only two types of ports. Input ports are used to receive input values from outside of the module, while output ports generate new values.

Only ports of the same type can be connected to form a channel. To establish a channel is to connect an input port of an module to an output port of another module. Since there is no specific identification of source or destination modules in a program, realization of a networking configuration can be delayed until the system execution, to be dynamically determined according to resource availability. PVM has no such abstraction of module, data, and connection. Every send/receive operation must explicitly specify the source/destination task ID, although the ID is also a logical representation of a node in a virtual machine.

4. Level of Integration. AVS is an integrated software development framework under which coarse-grained modules of different modules on a distributed system can be equally combined in the pipeline to solve a complex problem. Message passing in the framework is restricted in a dataflow fashion and must be coarse-grained. Modularity and many other conventional software engineering features such as reusability, object-oriented design, extensibility, software sharing, productivity, and rapid prototyping can easily be achieved in the AVS environment. It can be a generic integration framework to be extensible to parallel and distributed environment. Fortran-M attempts to be an extended parallel programming language to support the integration of task and data parallel modules at a language level. It can be either coarse-grained or fine-grained, but the most natural and efficient programming in Fortran-M is a coarse-grained task program to embed fine-grained data-parallel modules. PVM is a message-passing library that can be used to develop integration tools. It is the most flexible of them and provides fine granularity, yet it is basically a set of primitive routines and an execution runtime for explicit messagepassing programming. No modularity is assured and almost everything related to higher-level software engineering facilities has to be built from scratch.

2.4 Application Visualization System — A Testbed

for Software Integration in HPDC Applica-

tions

AVS [1] is a software development toolkit based on a dataflow model for scientific data visualization and process control. It incorporates visualization, graphics, visual programming, process management, and networking into a single comprehensive software development environment. We are most interested in its software integration capability.

AVS dataflow is a model of parallel computation in which a flow network of autonomous processes computes by passing object data along arcs interconnected by input/output ports. Each module, represented as an individual UNIX process, fires autonomously as soon as any requires new input object arrives on its input ports. It then produces an output object that flows on its output port, thus triggering further computations. A set of base communication objects is predefined, such as numbers and fields (n-dimensional arrays), and a collection of computational modules is provided that transforms sets of input objects into sets of output objects. AVS itself offers an extensive library of ready-to-use modules primarily meant to be used for graphics programming and filtering, as well as for specifying the protocol for developing and including user modules into the system. Specific applications are assembled from reusable modules with the aid of visual network editing tools. Modules are visualized as nodes, ports as terminals, and communication objects as links of an interactively constructed dataflow network.

Both process control and data transfer in AVS, are in a modular fashion and are completely transparent to the programmer. AVS provides a data-channel abstraction that transparently handles module connectivity and ports type-checking. The module programmer needs only to define the input and output ports in AVS-predefined data types and use a set of AVS routines and micros. Message passing occurs at a high level of data abstraction and only through the input and output ports. Each AVS application includes an AVS kernel and a set of modules specified by a dataflow network. An AVS kernel executes network specification and instantiates module processes as well as lower-level intermodular communication links based on the RPC/XDR UNIX network protocol. Another important feature of the AVS execution model is that it supports a remote module execution in which various nodes of the computational graph can be executed on distributed machines on a local or wide area network.

Integrating Multiple Parallel

Programming Paradigms

An integrated software environment that supports different programming paradigms and languages is important for solving a complex HPDC problem on a meta-computer. In this chapter we investigate how parallel programs written in different and multiple parallel programming models can be combined under the proposed single integration framework to solve an application.

By viewing different parallel programming paradigms as an essentially heterogeneous approach to mapping “real-world” problems to parallel systems, we discuss methodologies for integrating multiple programming models on massively parallel and distributed systems. Using a dataflow-based integration model built in the AVS testbed, we describe a simple, effective, and modular way to combine sequential, data-parallel, and explicit message-passing modules into an integrated parallel programming environment. Three case studies are chosen from numerical advection modeling, computational electromagnetics, and financial modeling, respectively, to illustrate our approach in multi-paradigm programming integration.

3.1 Introduction

A major issue in parallel processing is homogeneous versus heterogeneous at both software and hardware levels. There are currently two mainstream approaches in parallel software development to deal with this issue. One attempts to hide the heterogeneity of architectures from the programmer. This approach usually employs software technology, especially advanced compiler techniques, to present users with a homogeneous, high-level programming model. This methodology leads to highly portable programs across multi-platforms of different architectures and programming models and it greatly simplifies programming on parallel machines. However, the compiler design is extremely complex, and it is difficult to obtain efficient programs for general application on a particular machine. Examples of this approach include the FortranD project by Northeast Parallel Architectures Center at Syracuse University and Rice University [18], various automatic tools, and the efforts of HPF [22] and the emerging MPI message-passing standard [13]. The other approach acknowledges heterogeneity at both software and hardware levels by extending existing programming languages and software environments into parallel environments, thus exploiting the high performance of underlying architectures. The advantage of this approach is the resultant highly efficient programs and an open software environment flexible enough to support the integrated heterogeneous processing environment proposed in Chapter 2.

While the first approach is promising and represents a major direction in parallel programming language development, we believe that the future of high-performance computing lies in the integration of existing and emerging parallel architectures and technologies into a heterogeneous networked computing environment, able to solve general classes of applications with various requirements in scientific computation, communication, programming models, hierarchy memory, I/O, data processing and visualization. Another advantage of this approach is its ability to adopt existing programs in which only computationally intensive components need to be rewritten to run on supercomputers. This is especially important in porting a large existing application system when a significant proportion of the sequential program are “interface or management codes,” i.e., program segments that do not perform real scientific computational tasks but simply manage various system resources such as standard I/O, graphical user interface, system control interface, OS or file system interface, and networking interface, to name a few. This heterogeneous approach, together with our view of parallel software as a mapping process between two complex systems — problems and parallel computers [17] — would require integration methodologies and tools to facilitate multiple parallel programming models in an unified environment. Software integration is expected to play a critical role in parallel software development and applications.

3.2 Multiparadigm Parallel Programming Environ-

ments

For this thesis we shall consider that a programming paradigm is a model or an approach employed in solving a programming problem with a restricted set of concepts. Programming paradigms either induce or are induced by a class of languages, the programming environments in which these languages are supported, or the software engineering discipline one uses to produce systems within these environments. Typically, each class places different requirements on its associated programming environment [21].

3.2.1 Multi-paradigm Programming in Mapping a Problem Domain to a Parallel System

On existing parallel systems, there are two dominant parallel programming paradigms:

1. Data parallel, in which more or less the same operation is performed on many data elements by many processors simultaneously. The data parallel program exploits parallelism in proportion to the quantity of data involved, hence it offers the highest potential for concurrency. It is also easy to write, understand, debug, and port.

2. Explicit message passing, which allows two or more operations to be performed simultaneously. A message-passing program is free to arbitrarily distribute program tasks and data, using synchronization and data communication between nodes when necessary. The message-passing paradigm allows the programmer maximum flexibility and control in exploiting domain decomposition and in controlling decomposition of the application problem so that maximum efficiency utilizing an MPP hardware can be achieved through careful design of explicit message-passing parallel algorithms.

Like the trade-offs among various conventional programming languages, expressiveness and efficiency are the two major considerations in different parallel programming models. The trade-off is even more significant, because the driving force using a parallel computer is the performance requirement, and programming on parallel machines tends to be more difficult and less portable. Programming languages for massively parallel processors must go beyond conventional languages in expressivity and the cost/performance ratio. Unfortunately, unlike sequential machines and due to efficiency considerations on a particular SIMD/MIMD architecture with a unique interconnection topology, most parallel systems support only a single programming model, i.e., either data-parallel or message passing. Given a targeted machine and an application problem, usually an early (long before the implementation stage) decision must be made as to which parallelism should be used to solve the problem. This is not natural and should be avoided, as (especially for large application problems) some components may be well suited to the data parallel paradigm, while others are suited to the message-passing paradigm. We should exploit the parallelisms at each level of computation. There has long been a difference of opinion in the parallel-computing community on the issue of data-parallel vs. message-passing programming, and SIMD vs. MIMD machines.

Figure 3.1: Computation and simulation as a series of mappings

As studied in [17], by viewing “real-world” problems and parallel computers as complex systems, the fundamental issue in solving a problem on a parallel computer becomes whether we can easily and efficiently map a problem structure to a particular parallel architecture. A general mapping followed by a computer simulation typically consists of several mapping stages (Figure 3.1). We are interested in the software environment and programming models used to effectively map a large application onto a massively parallel system.

Parallel systems are built to solve large-scale, computationally intensive applications (e.g., “grand-challenge” problems). The broad range of HPCC applications may embody a rich diversity of program structures. It is anticipated that no one parallel programming model will satisfy all needs. Data-parallel, task-parallel, and objectparallel models have all found applicability to end user problems [27, 12]. A useful parallel programming environment for MPPs must incorporate semantic constructs for delineating program parallelism, identifying locality and specifying data set partitioning, as well as fulfil the normal requirements of conventional software engineering. It must permit the user access to a mix of parallelism models, either by incorporating them into a single schema or by providing the means for a single application to use different languages or different parallelism models for separate parts of a problem.

Figure 3.2: Integration of multiple programming models as a single mapping

In either case, a parallel software system will of necessity support interoperability among independently derived program modules, perhaps even running on separate computers in a distributed heterogeneous computing environment. This is especially important for irregular and dynamic problems and for multidisciplinary applications, as it is often inevitable that those applications require coordination or composition of multiple paradigms programming (e.g., high-level data-parallel and low-level explicit message passing), considering the complex of problem structures to be expressed and solved by the software environment. Such an example of a multi-model earth system in which an ocean model and an atmosphere model are loosely coupled is given in [15].

While the homogeneous approach emphasizes a virtual programming model due largely to portability consideration, and high-level and low-level software are used in a strict successive manner in the mapping shown in Figure 3.1, it is most likely that a large application problem would require programming models of different abstraction levels to be used in a hybrid fashion. For instance, Figure 3.2 gives another picture of how subtasks of an application task are mapped to a parallel system in a single mapping stage by multiple programming models.

A problem task may consist of subtasks requiring a set of computing services such as parallel computation/communication, parallel database management, parallel or sequential I/O, real-time visualization, sequential event-driven graphical user interface, networking interface, etc. This mapping process will require not only a “universal architecture” of the parallel computer to be able to efficiently support multiple programming models, but also a software environment capable of supporting task-level computation/communication and of integrating different programming paradigms into a single framework. Software integration techniques are clearly required when interfacing data and control flows among components (modules) of different programming models in this single mapping process.

The CM5 is one of today’s few parallel computers that support both data-parallel and explicit message-passing programming models. We choose it in the next sections to evaluate the methodologies of multiparadigm integration in this chapter. However, we believe that many of the issues discussed are also applicable to programming integration in a heterogeneous computing environment in which multiple parallel systems are connected by high-speed LAN and/or WAN networks.

3.2.2 Multiparadigm Programming

In this section we first examine integration techniques for multiparadigm programming on a sequential machine and then corresponding implementations under a parallel context such as on a Connection Machine CM5.

On a conventional uniprocessor machine, multiparadigm integration can in general be divided into the following four classes, based on the interface of how one paradigm is combined with the others:

1. Full integration. A single language has constructs supporting multiple programming paradigms. The transfer of program control and data among different language components at runtime is implicitly arranged by the compiler. The semantics of interfacing different paradigms is built into the language specification. Usually, a programming language evolves along this line, with a single paradigm at the initial stage and new language components added as applications grow and different paradigms are required. C++, while evolved from C (which in turn evolved from Pascal) now supports imperative, modular, and object-oriented programming paradigms and can be thought of as an example of this category. Another example can be found in [6] in which functional and logic programming paradigms are integrated into a single language. Among those using this approach in a parallel programming language, CC++ [2] and FortranM [14] are examples that have attempted to integrate into a single language framework different paradigms of parallel programming such as dataparallel, task-parallel and object-parallel paradigms, imperative and declarative programming, shared memory, and message-based programs. This approach provides the most flexible and efficient hybrid integration of multiple models. However, new languages have to be developed, and a complex semantics for interfacing multiple programming paradigms is usually inevitable. On an MIMD parallel system like the CM5, because data-parallel and messagepassing paradigms have dramatic implementation and performance differences in data mapping/processor allocation, task decomposition, and execution synchronization, and require unique semantics inherently in both the programming language and underlying system architecture, this integration approach may incur interface overhead and unexpected complex semantics. It therefore does not seem feasible on a distributed-memory machine.

2. Embedded integration in master-slave style. The interface is a built-in construct (or convention) in a master language to allow seamless invocation of subroutines (functions) written in a slave language. No global data can be shared directly by both languages, even though they have the same addressing memory space. Data transfer among modules in master and slave languages is identical to that in a single language and is carried out by passing pointers to the same shared memory so that there is no data copy cost. Separate compilations of different languages are required and their object programs are finally linearly linked together. The executable program runs as a single process. This approach provides a simple, coherent, yet efficient way to support integration at the language level. Among examples of this approach are inter-language calls on most conventional platforms, such as C calling Fortran77 or assembly, or vice versa. This is the common approach on most SIMD and MIMD machines in order to support the extension of parallel programming in host sequential languages, such as the CM5 calling CMfortran, C*, or CMMD message-passing primitives from Fortran77 or C.

3. Loosely coupled and process-based integration. Different individual program modules written in the same or different languages are compiled and linked separately, while their executables run concurrently as multiple processes time-sharing a CPU. Message passing among program processes is normally via socket-based interprocess communication (IPC), although if two modules share a host it is common to use shared memory to optimize the communication. Runtime process control is coordinated by a supervising manager (or daemon) or the operating system. Currently, most windowing systems and networking applications (for instance, the client-server model) follow this approach. On the host machine of a parallel system, IPC-based systems provide an open environment for parallel programming software to be naturally migrated into the conventional software environment in which software such as the concurrent operating system, networking, graphics, I/O, and sequential programming languages have been well established for many years. This process-based approach also opens great opportunities to construct a “meta-computer” via a networked environment [25]. Depending on the method (protocol) used to facilitate the IPC (such as data exchange, management of buffers and sockets, threading, process control and synchronization), an integration system of this kind can be quite different. For example, the PVM [73], a process-based general message-passing system, provides a set of user interface primitives that may be incorporated into existing procedural languages. In our work we use AVS [30], which is our testbed of the proposed process-based integration framework. We will demonstrate in later sections the advantages of the AVS’s dataflow IPC model in facilitating data-parallel and message-passing programming paradigms on the CM5.

4. Not-at-all integration in non-interactive style. Data passing and sharing among different program modules, usually stand-alone executables, are carried out at the (UNIX) operating system level through shell scripts, pipes, intermediate data files, and other pre- or post-processing techniques. This is the usual way for an application to use collectively the (parallel) computing services when it does not require their programming tasks to run in an interactive way. this approach incurrs a high system overhead.

3.3 A Dataflow-Based Integration Approach for

Multiparadigm Programming

We choose two system configurations for applying the proposed integration framework to multiparadigm programming support. One is on an MIMD parallel system Connection Machine CM-5, and the other is on a heterogeneous computing system where an IBM SP2 and a DEC Alpha workstation cluster are interconnected via a LAN. In the first configuration we describe the techniques used to create a general programming environment that couples data-parallel and explicit message-passing programs in a unified framework. The second one deals with system integration issues to allow fine-grained PVM programs to be seamlessly embedded in coarse-grained AVS visualization modules in a loosely distributed environment. Several key functional components, including parallel and distributed computing, interactive remote userinterface, advanced data visualization, and transparent networking are combined to support real-time simulations over a heterogeneous computing and networking environment.

Figure 3.3: A general AVS/CM5 integration model for multiparadigm programming

3.3.1 A General Parallel Programming Environment on a CM5 with AVS

Appendix B.1 describes the parallel programming available on the CM5 and briefly examines the various integration approaches currently used in the CM5 system.

Although AVS does not provide support for parallel computing within a module, a specific module may use machine/language specific features to achieve parallelisms such as data parallel or explicit message passing. Using the modular process management in AVS and the time-sharing CMOST on the control processor, together with CM5’s parallel programming software, we can naturally combine programs in the data-parallel, message-passing, and sequential programming languages available on the CM5 into a single process-based software system. As shown in Figure 3.3, an AVS/CM5 integration framework has two distinct levels of programming interfaces. At the higher level, an AVS dataflow model is used to express and implement data and control flows among various modules that represent decomposed subtasks of a specific application. Intermodular communication is handled by AVS IPC protocol. Process control (e.g., synchronization and flow control), scheduling, and realization of interactive visualization at module level are also handled at this interface and are supervised by the AVS kernel. All the sequential process threads at this level are executed on the same control processor of the CM5. The AVS kernel can be on the same CP or on a different machine networked to the CP. At the lower level, a specific CM5’s parallel programming model (for example, a CMFortran subroutine or a CMMD program) is used to implement a highly parallelized algorithm in fine-grained parallelism, best suited to solve the subtask represented. All the parallel process threads at this level are executed on the CM5’s processor nodes, and the CM5’s specialized data network and control network (i.e., the fat-tree topology) are used for communications among the processor nodes within a module. Data transfer and thread control between the two levels in a process is handled by the host-node protocol in CMOST. An AVS/CM process can be a module or a set of modules written in the same language/paradigm. A typical integrated parallel module will have two distinct threads at the two different levels to be executed on different physical processors of the CM5 system (i.e., CP or PNs). They are represented as two different portions in the program code of a module:

1. AVS/CM interface part. This is a sequential code in Fortran77, C or C++ to coordinate data flow between AVS modules and to control flow between the AVS kernel and the CM parallel system. It contains code declaring the AVS module’s input/output ports and (graphical) parameters. For a data-parallel module, it has program segments for packing input data, received either from other AVS modules or by sequential subroutines in the same module into parallel data, e.g., to transfer serial arrays into data parallel arrays or decompose/load block data onto each processing node, or vice versa. For an explicit messagepassing module, it is a host program for the CMMD host/node programming model.

2. CM computation parts. These data-parallel subroutines in CMFortran or C* — or CMMD node programs in Fortran77, C, or C++ — to perform main calculations requiring intensive computation and communication. This part of the module is almost identical to a stand-alone parallel program implementing core parallel algorithms for a subtask (if not integrated under the AVS framework), except that it becomes a parallel subroutine invokable from the AVS/CM interface part.

After both parts of a module are compiled separately, possibly with different compilers, they are linked into a single executable. Multiple modules can either be compiled and linked individually into separate executables (processes) or they can be linked into a single executable. Modules in different processes must communicate data through the CP under the supervision of the AVS kernel. If the modules are written in different paradigms (e.g., CMFortran or CMMD), they are in different processes. If both modules are in the same process on the CM5, only a pointer needs to be passed; therefore, modules written in the same paradigm can be grouped in a single process and data transfer among them has no data-movement cost.

This integration framework provides a sufficiently high-level way to couple modules written in multiple programming paradigms into a single environment on a CM5, so that programmers have the freedom to choose either parallel programming method to solve their problem, independently or in a mixed way. This is a general parallel software engineering environment that allows more than one programming paradigm to be used, including sequential and parallel models, data-parallel and explicit message-passing programming, concurrent and pipeline execution, and task and functional parallelism. Running in an interactive environment, it gives the user maximum control over the optimization of heterogeneous resources and capabilities of a high-performance computing system. Other advantages using this integration system include:

1. Modularity and template-oriented. Programs are constructed by using explicitly declared communication channels to plug together program processes.

A process can encapsulate common data decomposition, manipulation and internal communication, and the programming paradigm inherent in the code. The modular approach of AVS also lends itself well to software sharing, module reuse, extensibility, and flexibility.

2. Module reusability. CM5 data-parallel or message-passing codes and algorithms are modularly developed and compiled once and can then be reused indefinitely in a plug-and-play mode in a variety of diverse scientific applications. Because the interfaces between modules are well defined, modules developed at different computer sites by different developers and in different paradigms may be freely shared. Modules created for one application can readily be used in another similar application with little or no change.

3. Flexibility and extensibility for rapid prototyping. At the first stage of an application development, programmers can concentrate on task decomposition, parallel algorithm design, and performance improvement and totally ignore system integration issues. They may write their own modules in their favorite programming paradigm and language binding. The programs are then migrated to the AVS environment by simply attaching and linking the AVS interface code, while the main parallel algorithms can remain untouched.

4. Hierarchical parallelisms. At the top of the task level decomposition of the problem domain, modules that are connected in the dataflow programming paradigm have the potential to run concurrently or in a pipelined way. At this coarse-grained process level, functional parallelism can also be achieved such that graphical interface, sequential and parallel I/O, remote networking access, and the decomposed task computation can be carried out concurrently by different components of the CM5 supercomputer. The second level of the hierarchy is the hybrid data parallelism in data-parallel modules and control parallelism in explicit message-passing modules. At the bottom level, parallelism can be thought of as inherent in the pipelined processing hardware, as are the vector

Figure 3.4: The parallelism hierarchy in the AVS/PVM system

units in the CM5.

5. Safety. Operations on communication channels are restricted so as to guarantee deterministic execution. Channels are strongly typed, allowing a compiler to check for correct usage.

3.3.2 Integration of PVM Programs into AVS Environment

This section briefly describes how fine-grained parallel modules in a portable messagepassing system such as PVM can be naturally incorporated into the coarse-grained dataflow modules in AVS. Background information about the Parallel Virtual Machine (PVM) can be found in Appendix B.2.

By integrating PVM software into the AVS framework, the resulting system offers an equally sophisticated networking and visualization functionality with integrated networking and visualization programming interfaces.

There can be two basic approaches to incorporating PVM programs into an AVS framework. They are based on the ways in which an AVS kernel interacts with PVM daemons:

1. Our first approach embeds a PVM programming model at a lower level into AVS dataflow communications at the top inter-module level. As shown in Figure 3.4, there is a parallelism hierarchy in the system at two distinct levels: dataflow parallelism occurs among AVS visualization modules and between an AVS module and a remote PVM (host) computation module, and a general message-passing paradigm is used by the PVM node tasks. A PVM node task is spawned from the AVS/PVM host module when the module is registered in the Network Editor. The AVS kernel communicates with only one PVM host daemon through AVS input/output ports. All the AVS programming features (such as transparent networking, modular process management, and event-driven dataflow) remain intact, as well as the PVM concurrent programming paradigm. We use this approach in our case study, discussed in Appendix B.4.

2. In the second approach, within the data-flow execution of AVS modules PVM primitives are used by AVS remote modules to perform temporal synchronization of arbitrary patterns or even transfer data among subroutine or coroutine modules. The major purpose of PVM here is to enhance remote process concurrency control in the AVS kernel. In this approach a remote AVS/PVM module consists of an AVS/PVM host part and a PVM node part. The module can arbitrarily communicate with other AVS/PVM modules, a host part or a node part, or even both. This communication can occur anytime before or after AVS modules transfer data through normal dataflow input/output ports. Each AVS/PVM module executes as a relatively autonomous process (or a group of loosely coupled processes) similar to an AVS coroutine. The AVS kernel can communicate with as many PVM daemons as required. This approach requires that the existing AVS kernel’s interface to the Network Editor be extended to allow a spawned AVS/PVM module from a module program to be able to register in the Editor and to make port connections.

3.3.3 Limitations in the AVS Integration Model

The process-based integration system offers many advantages for a parallel software engineering environment in terms of multiparadigm programming, software reuse, and modularity. However, towards an efficient and fully functional multiparadigm system, the current AVS framework suffers a few significant performance problems and operational restrictions.

1. Intermodular Communication. The major performance problem is a centralized AVS kernel versus a decentralized parallel system. Restricted by the kernel and its IPC data-flow protocol, any communication between two parallel modules has to go through the sequential host-node channel and the modulemodule pipeline on the control processor. Figure 3.5 illustrates this with two modules (m_{1} and m_{2}), each with two separate but linear threads (h_{1},h_{2} on the CP and p_{1},p_{2} on the PNs). When m_{1} needs to communicate to m_{2}, it has to follow the serialized data transfer path p_{1} ⇒ h_{1} ⇒ h_{2} ⇒ p_{2}, while the most efficient path (or parallel channel) is directly p_{1} ⇒ p_{2} on PNs using CM5’s highbandwidth data and control networks. In this sense, the current AVS kernel serializes parallel channels. This may not be a performance bottleneck between a data-parallel module and a message-passing module, as in most cases scalar data are expected to flow between modules in different parallel paradigms. This is not true for modules in the same paradigm (e.g., CMFortran modules) between which, inevitably, very large scientific data sets must be transferred. In addition, because each module is buffering its entire body of data on the input and output ports and all the intermediate data have to reside in the memory of the CP, the total memory requirement on the CP can be very large. Continuous memory-disk swapping of processes by the operating system can make this situation even worse.

2. Dynamic Flow-Network Reconfiguration and Load Balance. The current AVS system provides very limited support to dynamic modification of a flow-network, e.g., processes and their communication requirements may change Figure 3.5: Serial AVS I/O vs. parallel node-node channels for intermodule communication

with time — processes can be created or destroyed, or a communication pattern may move at runtime [26]. It is also difficult in AVS to perform fine-grained dataflow paradigms such as a round-way dataflow, i.e., an output port of a downstream module in reverse is connected to an input port of a upstream module. In certain “real world” multitasking applications the need arises to create or (re)initiate individual modules in a dynamic fashion. Consider, for example, a signal-processing component of a command and control system. The input dataset typically comes from several sensors, is processed by a set of coupled filters, and is then analyzed by a set of higher-reasoning modules. A particular specialized down-stream filter might be requested and activated in runtime by the upstream reasoning module. Such a transient specialized module could not be visually pre-edited in the static AVS flow-network, since it is invoked dynamically in a data-driven fashion. A dynamic reconfiguration of a flow-network is also required by load balance issues. The AVS/CM5 system assigns modules to processor nodes in a static fashion and offers no support or visual metaphor for task mobility. A good example is provided by an adaptive multi-grid PDE algorithm operating on a highly nonhomogeneous manifold. Such an algorithm would start as a single data-parallel task and would often spawn additional tasks in runtime. For instance, a complex local geometry might be detected that requires a high-density local mesh refinement. Local nonhomogeneities of this type will be naturally load balanced by dynamically distributing such spots, thereby spawning new concurrent data-parallel PDE tasks.

3. Synchronization Control and Scheduling of Module Processes The only synchronization mechanism in the AVS model is passive causality provided by the data-flow paradigm. High-performance interactive applications often require near real-time operating system support and tight synchronization of various media streams. An extension of the dataflow model is required to handle such requirements. This includes both the underlying low-level messagepassing system and the high-level, scriptable and visual network editing tools. In the current model the only visual metaphor is that of a graph connectivity, implemented in terms of nodes-modules and links-communications. Suitable objects and visual tokens need to be included to encompass the synchronization metaphor and to provide graphical tools for representing priorities, threads, and synchronization objects such as semaphores, mutexes, stacks, FIFOs, or message queues.

The performance of these systems on massively parallel machines will in large part depend upon the systems’ capability to support high bandwidth and low latency for data transfer between a host machine and all the node processors, and the way in which a problem’s tasks are decomposed. Task granularity must be restricted to a coarse-grained domain, and task partitioning should be carefully evaluated to consider the balance of each module’s computation time and process communication time. System software is needed to support high-bandwidth parallel I/O and sharedmemory segments between processes. The CM/AVS system developed at Thinking Machines is expected to greatly reduce the process-level, data-movement overhead [23, 61]. In the CM/AVS, CM5 shared-memory regions are used to exchange data if they are in the same CM5 partition; they exchange data directly over the routine network via a CM-domain socket if they are in different partitions. They may communicate using parallel sockets, with serial modules on a workstation over the fastest available network connecting the various components.

3.4 Case Study I — Comparison of Numerical Ad-

vection Models

The first case study described in Appendix B.3 is an earth science application. Using the process-based framework, we developed a real-time simulation prototype in which multiple numerical advection models written in different programming models are coupled on a CM5 and their simulation results graphically compared through a graphical user interface.

3.5 Case Study II — Option Price Modeling

The second case study is a prototype simulation-on-demand system in which parallel option-pricing models are locally implemented on two distributed systems, an Ethernet-connected IBM SP2 and a FDDI-based GIGswitch-connected DEC Alpha farm, and then coupled with an interactive graphical user interface over an ATMbased wide area network. This integrated networking/visualization framework allows us to use a high-level software approach to combine AVS visualization modules with remote concurrent PVM tasks in fine-grained parallelism. Details of this case study are in Appendix B.4.

3.6 Case Study III — Distributed Computational

Electromagnetics Systems

In the last case study we describe the development and implementation of a computational electromagnetics application on multiple parallel and distributed systems. This application requires the decomposition of problem tasks into different tightly coupled program components to be written in different parallel programming models in order to achieve an optimized balanced performance when running on a targeted meta-computer. We also describe a framework that embeds a resource scheduling component into the proposed dataflow integration model.

The complete application is given in Appendix B.5.

3.7 Conclusion

The purpose of integrating multiparadigm programming is to provide programmers with a rich and attractive programming environment in which a choice of high- and low-level programming models can be made for problem structures, as well as particular machine architectures, to facilitate a gradual and eventually efficient mapping from problems to parallel systems. In the case where multiple programming languages are required, this environment supports a seamless and modular way to interface the data and control transfer among processes in different languages.

With its visual programming interface, modular program structure, dataflowbased execution, interactive visualization functionality, and its open system characteristics, a process and dataflow-based integration software like AVS provides an excellent framework with which to facilitate the integration of the various system components required by large, multidisciplinary applications, including the integration of sophisticated interactive visualization, database management, heterogeneous networking, massively parallel processing, and real-time decision making. It is also a useful tool for software development and project planning. We believe that with the adoption of HPCC technologies into industry applications, system integration will CHAPTER 3. MULTIPARADIGM PARALLEL PROGRAMMING 67 play an important, ever-growing role in the development and system design of parallel software.

Integrating Data Visualization and

Parallel Computations

Network-based concurrent computing and interactive data visualization are two important components in industrial applications of high-performance computing and communication. In this chapter we apply the proposed execution framework to build interactive remote visualization systems for real-world applications on heterogeneous parallel and distributed computers. Using the dataflow model of the AVS testbed in two case studies, we demonstrate a simple, effective, and modular approach for coupling parallel simulation modules into an interactive remote visualization environment. The applications described in this paper are drawn from our projects in financial modeling and computational electromagnetics for industry.

4.1 Introduction

Scientific visualization has traditionally been carried out interactively on workstations, or in post-processing or batch on supercomputers. With advances in highperformance computing systems and networking technologies, interactive visualization in a distributed environment becomes feasible. In a remote visualization environment, data, I/O, computation and user interaction are physically distributed through high-speed networking to achieve high performance and optimal use of various resources required by the application task. Seamless integration of high-performance computing systems with graphics workstations and traditional scientific visualization is not only feasible, but will be a common practice with real-time application systems.

Advances in parallel computing systems and network technology provide new opportunities for implementing computationally intensive applications. Applications that integrate modeling and simulation with large-scale information processing often require an interactive graphical user interface (GUI) in a real-time computing environment. Most GUIs are event-driven and serial in nature, making them unsuitable for parallel implementation. On the other hand, visualization tools for parallel systems often require special hardware support or are developed for a specific hardware/software system and are not portable. Parallel computing environments of the future will likely be based on a heterogeneous, networked computing system and will be required to solve applications with various requirements in scientific computation, communication, programming models, memory hierarchies, I/O, database management, and visualization. Network-based concurrent computing and advanced data visualization are two important components in real-world HPDC applications, especially for industrial problems requiring real-time simulation/modeling, information processing, and system integration.

In several real-world HPDC application projects we have developed real-time parallel numerical simulation/modeling applications in the areas of financial modeling [4, 7], computational electromagntics [5], computational chemistry [37], and environmental modeling [6]. These applications use a range of parallel and distributed systems from MPP machines that include the CM2, CM5, DECmpp-12000, and IBM SP2 to networked heterogeneous workstation clusters such as the DEC Alpha Farm and coupled MPPs and workstations. These projects are part of the major HPDC technology transfer program, InfoMall [20], developed at NPAC, that consists of a set of interactive distributed high-performance information systems collectively called InfoVision [19] (INFOrmation, Video, Imagery, and SImulation ON-demand). Targeted at state-of-the-art applications and enabling technology on NYNET (a regional ATM-based WAN network) and the utilization of the supercomputing resources at NPAC, InfoVision focuses on real-time information production, analysis, access, and integration on distributed and parallel computing systems. This allows geographically located regional end-users to interactively access multimedia information and to remotely perform parallel simulation demonstrations on supercomputers over the high-speed WAN network, NYNET.

As a core-enabling technology in simulation on-demand applications, the interactive software integration environment plays a central role in coupling parallel and distributed computing, interactive remote user-interface, advanced data visualization, and transparent networking to support real-time simulations over a heterogeneous computing and networking environment.

In this chapter we describe our approach to building interactive visualization systems for the InfoVision simulation-on-demand projects. We use AVS as the integration framework testbed to facilitate both networking and scientific data visualization in all the case studies. This chapter is organized as follows: In Section 4.2 we briefly review some related work in embedding parallel applications within the framework of modular distributed visualization systems. In Section 4.3 we detail a distributed visualization and computing framework built on the proposed dataflow model and to be elaborated on in later sections when it is applied to specific applications and parallel systems. Section 4.4 describes a prototype visualization system for a financial option-price modeling application originally implemented on a CM5 and a DECmpp-12000 and ported to an IBM SP2 and a DEC Alpha farm. A remote visualization environment for an electromagnetic scattering simulation on a CM5 and networked workstations is given in Section 5 to demonstrate the effectiveness of this framework. Finally, we state our conclusions in Section 4.6.

4.2 Related Work

Research and applications in distributed visualization environments have been very active in the HPCC community since the early 1980’s. Interactive visualization is identified as a critical component in the national HPCC initiatives [58, 59]. Many efforts are being made toward developing interactive distributed visualization applications on parallel systems in the framework of the AVS dataflow model [62, 63, 64, 65, 66]. Most efforts, however, are focused on customized solutions in which front-end visualization/graphics components are tightly coupled with back-end parallel engines where specialized graphics hardware is usually required to support high-end graphics capability. To be able to support a general interactive software development environment for HPDC applications, our approach is focused on a loosely coupled visualization and computing framework portable to different distributed computing system configurations.

Similar work in this direction includes CM/AVS and MP/Express. CM/AVS [23, 61], developed by Thinking Machines, is a modular visualization system in which AVS is used as the front-end visual programming interface for scientific data visualization. It is tightly coupled with the CM-5 parallel supercomputer for back-end parallel computations. The major difference between CM/AVS and the distributed AVS environment is that instead of conventional TCP/IP sockets and UNIX-shared memory, specialized CM-5 shared-memory regions or CM-domain sockets are used in CM/AVS to exchange data among distributed parallel modules on CM-5 processor partitions.

MP/Express [60], a research project in AVS, Inc., is an enhanced AVS/Express for building parallel and distributed AVS applications with a visually-based software development tool. It incorporates a middle communication layer between the AVS kernel and application modules such that distributed modules can communicate with each other using the portable message-passing layer and a data layout system for automatic data distribution among multiple node processes of a distributed module in an SPMD (Single Program Multiple Data) paradigm. The serialized data transfer channels between parallel modules found in AVS become parallel channels in MP/Express, and an automatic data layout for splitting and recombining data sets in AVS data types is available to module developers.

4.3 A Distributed Visualization and Computing

Model in AVS

4.3.1 A Dataflow-Based Visualization and Computing Model

In a typical remote visualization environment a simulation cycle is started from a GUI module running on a local machine, with simulation parameters represented as slide buttons and dials that provide real-time instrumentation of the simulations’ progress, as well as simulation steering (changing simulation parameters before a new simulation cycle). Source data for the simulation can also be read in by the GUI module from databases or data files. The computational task of the simulation is decomposed into several computationally independent subtasks that are represented as computing modules distributed to different remote machines. Input data are collected from the GUI by user runtime interaction or from disk files, or from both, and broadcast to computing modules on the remote machines. There is little or no data transfer among modules on different remote machines. A remote machine may be a workstation or a supercomputer, whichever architecture and computational power is best suited to the decomposed subtask. The simulation ends with some kind of GUI rendering/viewing modules that run on the local machine and use results generated from remote modules. Fundamentally, this is a dataflow (data-driven) programming model in which the activation of a module process is triggered solely by the availability of input data from another module process on either the same or a different machine.

This general model is well suited to rapid-prototyping for certain simulation and modeling applications that require both scientific data visualization and highperformance computing. At the software environment level this model requires only the support of high-level data visualization and networking facilities. Most important, this data flow-based model is well supported by most commercially available data visualization environments such as AVS [1] from AVS Inc., Explorer [31] from Silicon Graphics, Inc., and Data Explorer from IBM.

As shown in Figure 4.1, using the dataflow programming model in AVS, message

Figure 4.1: A general visualization/computing model

passing among modules on the same machine and different machines is identical and completely transparent to a module programmer. The AVS kernel (protocol) supervises data transfer, which is eventually carried out by TCP/IP at a lower level. The module programmer needs only to define module input and output ports in AVS predefined data types. Message passing among AVS modules occurs only through I/O ports and uses AVS’s IPC or RPC protocol. A set of routines for initializing and describing modules to AVS, as well as parameter handling, accessing data, error handling, and coroutine event handling, is provided. Data sources and destinations can flexibly be defined by visually connecting module input and output ports using a network editor.

Data transfers between host and node programs of a parallel module rely on either portable message-passing protocols in implementing a parallel module such as PVM and MPI, or on native protocols specific to a parallel system such as CMMD,

Figure 4.2: A general performance model

CMFortran on a CM5, MPL on an IBM SP2, or HPF on a DECmpp. They are strictly confined within a parallel module. Message passing between two parallel modules on different machines must go through the dataflow channel between the two sequential AVS modules embedded in the host parts of the parallel programs.

4.3.2 Performance Analysis

A generalized performance model of our networked system is shown in Figure 4.3.2. In one complete modeling cycle, starting with the broadcast of new data from M_{I}, and ending with renderings on M_{h}, let t_{calc} be the summed calculation time on all machines and let t_{comm} be the summed communication time in the system. Let T_{d}, T_{s} be the total time to complete such a cycle in the distributed system and in a single-machine system, respectively. Other symbols are defined as follows: c_{k} = communication time of required message passing from M_{I} to M_{k}; d_{k} = communication time of required message passing from M_{k} to M_{h}; m_{k} = calculation time of performing required computation on M_{k}; r_{k} = calculation time of performing required rendering by P_{k} on M_{k}; t′_{calc} = the total calculation time on a single-machine system; m′_{k} = calculation time of performing required computation on a single machine system.

In a single-machine system, M_{I} = M_{h} = M_{1} = M_{2} = ⋯ = M_{n}, t_{comm} = 0 and

Figure 4.3: Different cases of the pipelined calculations and communications

T_{s} = t′_{calc} = ∑_{k=1}^{n} (m′_{k} + r_{k}). In the distributed system based on TCP/IP protocol, we observe: sequential message passing from M_{I} to M_{k}, although the sending order can be scheduled; execution of computing process on M_{k} upon receiving a new message from M_{I}; and sequential rendering of process P_{k} on a single-processor machine M_{h}, where each P_{k} is activated only after receiving a new message from M_{k}, and no other P_{j}(j ≠ k) is running on M_{h}.

Calculation and communication among different machines in the system are pipelined. We can overlap the following processes: (1) c_{k} with m_{j}, d_{j}, and r_{j}; (2) m_{k} with d_{j} and r_{j}; and (3) d_{k} with r_{j}. Figure 4.3.2 illustrates this overlap of processes in a system of (n = 2) machines. For each machine M_{k}, the total time to complete a cycle is T_{d} = c_{k} + m_{k} + d_{k} + r_{k} + i_{k} , where i_{k} is the idle time of M_{k} in the cycle, excluding the waiting time r_{k} for the rendering on M_{h}. Thus, nT_{d} = t_{comm}+t_{calc}+t_{idle}, where t_{comm} = ∑_{k=1}^{n}(c_{k} + d_{k}), t_{calc} = ∑_{k=1}^{n}(m_{k} + r_{k}), and t_{idle} = ∑_{k=1}^{n}i_{k}.

The speedup S of a distributed system over a single-machine system is:

S = \frac{T_{s}}{T_{d}} = \frac{n}{\frac{t_{calc}}{t′_{calc}} + \frac{t_{comm}}{t′_{calc}} + \frac{t_{idle}}{t′_{calc}}} .

Speedup depends on four components: (1) n, the number of distributed computing modules; (2) t_{calc}/t′_{calc}, the ratio of total calculation time on the distributed system to calculation time on a single-machine system; (3) t_{comm}/t′_{calc}, the ratio of total communication time in the distributed system to calculation time on a singlemachine system; and (4) t_{idle}/t′_{calc}, the ratio of total idle time in the distributed system to calculation time on a single-machine system. As Figure 4.3 illustrates, for a given application on a given system (fixed c_{k}, d_{k},m_{k},r_{k}) t_{idle} depends on the order of messages sent from M_{I} to M_{k}, making this a scheduling problem. In the best case, t_{idle} = ∑_{k=1}^{n−1}c_{k}(n − k) + ∑_{k=1}^{n}a_{k}r_{k}, where {a_{1}, a_{2}, …, a_{n}} is a permutation of the set {0, 1, …, n − 1}.

4.4 Case Study I — An Interactive Visualization

Environment for Financial Modeling on Het-

erogeneous Computing Systems

Financial modeling represents a promising industrial application of high-performance computing. In previous work, parallel stock option pricing models were developed for the Connection Machine-5 and the DECmpp-12000 [10, 11] and were later ported on an IBM SP2 and a DEC Alpha cluster. These parallel models run approximately two orders of magnitude faster than sequential models on high-end workstations. To further develop this application in real-industry use, a portable, workstation-based, interactive visualization environment is developed for a heterogeneous computing environment.

Stock option pricing models are used to calculate a price for an option contract based on a set of market variables, (e.g., exercise price, risk-free rate, time to maturity) and a set of model parameters. Model price estimates are highly sensitive to parameter values for volatility of stock price, variance of volatility, and correlation between volatility and stock price. These model parameters are not directly observable and must be estimated from market data. Using optimization techniques for model parameter estimation holds great promise for improving model accuracy.

We use a set of four option pricing models in this study. Simple models treat stock price volatility as a constant, and they price only European (option exercised only at maturity of contract) options. More sophisticated models incorporate stochastic volatility processes and price American contracts (option exercised at any time in life of contract) [38, 39]. These models are computationally intensive and have significant communication requirements.

Analytic models are useful tools in the financial market, but require expert interpretation. To further evaluate and optimize pricing models to run in a realistic application environment, we combine high-performance computing modules for realtime pricing with real-time visualization of model results and market conditions, and a graphical user interface allowing expert interaction with pricing models. We envision a market expert using such a system to start and stop a set of models, adjust model parameters, and call optimization routines according to dynamically changing market conditions.

A detailed description of this case study can be found in Appendix C.1.

4.5 Case Study II — An Interactive Remote Visu-

alization Environment for an Electromagnetic

Scattering Simulation on a High-Performance

Computing System

Electromagnetic scattering (EMS) simulation is an important computationally intensive application within the field of electromagnetics and is an area of emphasis in the national high-performance computing initiative [16]. Advances in high-performance computing and the data visualization environment provide new opportunities to visualize real-time simulation problems, such as EMS, that require significant computational resources. In this case study an integrated interactive visualization environment was created for an EMS simulation, coupling a graphical user interface (GUI) for runtime simulation parameters input and 3D rendering output on a graphical workstation with computational modules running on a parallel supercomputer and two workstations. The AVS dataflow integration model was used as integrating software to facilitate both networking and scientific data visualization. We explore the dataflow methodology to naturally integrate data visualization, parallel systems, and heterogeneous computing. Major issues in integrating this remote visualization system are discussed, including task decomposition, system integration, and concurrent control — all under a high-level distributed programming model.

We describe the complete development and prototype system of this application in Appendix C.2.

4.6 Conclusion

High-speed networks, combined with advances in computational power, will support a new set of applications designed for high-performance distributed computing. In real-world HPCC applications, computational power alone is not sufficient. We see a need for a broader parallel software engineering environment, such as the interactive visualization system described in this chapter, which combines and takes advantage of high-performance computing and other well-established computing areas such as data visualization, computer graphics, system integration, and software engineering. We focus here on an execution model required to combine real-time modeling/simulation, parallel processing, visualization, and interactive user control.

With its visual programming interface, modular program structure, dataflowbased execution, interactive visualization functionality, and its open system characteristics, a process-based integration framework like AVS provides an excellent framework with which to facilitate integration of various system components into a large, multi-disciplinary application, including the integration of sophisticated interactive visualization, database management, heterogeneous networking, massively parallel processing, and real-time decision making. It is also a useful tool for software development. We believe that with the adoption of HPCC technologies into industry applications, data visualization and software engineering will play an important ever-growing role in technology transfer and parallel software development.

Integrating World Wide Web and

Distributed Computing

The World Wide Web (WWW) has become a familiar place and a popular way for corporate and Internet users to publish and exchange information over networks. Base components of the WWW include: HTML, the markup language; HTTP, the networking transport protocol; and HTTP servers and Web browsers. Both the concept and the mechanisms found in today’s WWW have been known and widely practiced in various software systems and applications before WWW was invented. These include the client-server models in networking, hyperlink-oriented markup languages in authoring (Latex, HTML, hyperlinked help systems, etc.), and a data-transfer protocol (TCP/IP, PVM, etc.). Indeed, many existing software systems and protocols such as FTP, Gopher, Archie, and News are naturally inherent in the WWW. We can view the WWW as a networking software integration framework that combines various software components into a single, unified software infrastructure and a new distributed computing paradigm for multi-media information systems and Internetbased applications. Based on this observation, in this chapter we illustrate how the general software integration model proposed in Chapter 2 can fit naturally into the WWW environment. In Section 5.1 we show that the WWW client-server model is essentially a special dataflow model in a networking software environment. We study issues in building general distributed computing applications on the WWW in Section 5.2. In Section 5.3 we examine a particular area of WWW application in integrating a relational database management system into the Web environment. The last two sections give detailed case studies in WWW software integration: Section 4 describes an Internet collaboration system built on the Web and a parallel Oracle RDBMS system; Section 5 is a WWW search system.

5.1 Introduction to WWW Client-Server Model

Client-server architecture has been a major execution model in networking and distributed systems. It is widely used in the computer industry, ranging from database access, file and printer sharing, and desktop windowing systems to information resource sharing/disassimilating. The World Wide Web adopted the open architecture of a client-server model for networking navigation and multimedia information concerning authoring and publishing. The fundamental architecture of the WWW is built on the client-server model.

As shown in Figure 5.1, major components of the WWW client-server architecture include:

1. Web clients. Each Web client in the WWW runs on a machine that is different from Web servers. It is usually represented by a single Web browser as the (graphical) user interface, while one or more underlying processes may be composed of the Web client program. The Web browser is not just an ordinary client program on top of the operating system, but a software interface on the local machine that provides an enhanced software engineering environment through which various local and global computing resources can be integrated to be present in front of the end-user. Major features of the Web client include:

(a) A portable and uniform GUI for networking applications, such as a hypertextbased document on authoring and publishing, multimedia presentation, and data-access to databases and other centralized resources. The GUI CHAPTER 5. WWW AND DISTRIBUTED COMPUTING 82

Figure 5.1: The WWW Client-Server Architecture provides an abstraction layer on the underlying operating system and communication that is independent of client platform and server locations.

(b) Desktop intelligence that can be built into client computation. A Web browser assembles client requests or queries from the GUI to be sent to a Web server and reassembles/analyzes replies or results received from the server. The emerging Web client technology (such as Java [105], Javascript, and plug-in) can embed sophisticated desktop intelligence and significant computation into the client program, either tightly or loosely integrated with the Web browser. For example, all data integrity validation in a client-server database application can be done on the client side.

(c) A source of originating requests in an event-driven fashion. The GUI on a Web browser is basically an event-driven interface. For instance, the event of a mouse clicking on a hyperlink generates a request to be sent to a remote Web server for service. Other non-event-driven applications can be accommodated, in which case the Web browser is merely a GUI front-end for display. In a typical Web client-server interaction, any request triggered by external input is originated from the client, which can be viewed as the source of the dataflow execution to be illustrated later in this section. Part of a Web client program is indeed a source module in our dataflow integration model. The event-driven model is fundamentally a dataflow model, commonly found in graphics and client-server applications.

2. Web servers. Conventionally, a Web server is a software program that runs on a remote host machine and listens to a particular network port to handle requests from clients via the HTTP protocol. If the request is for a file, the server simply locates the file and sends it back to the client. If the request requires more complicated processing on the server host machine, the server uses a CGI program to spawn additional processes to handle the client’s request. To a broader extent, a generic Web server can be an all-purpose software system that remotely handles application requests over a network.

3. Web client-server data transfer protocols. The WWW has become more than a single networking system. It is a networking framework into which various conventional or emerging network services and distributed software systems can be integrated. Depending on the Web clients and servers, Web client-server data-transfer protocols can be quite different and diverse. When a standard HTML browser interacts with an HTTPD server to gather remote HTML documents, the native data-transfer protocol, HTTP, is used. HTTP is designed for the rapid and efficient delivery of hypertext materials. HTTP is a stateless protocol with a single transaction per connection in order to be simple and to distribute multimedia data efficiently. “Get” and “Post” are the only message-passing primitives to accomplish the “send” and “receive” functions between client and server programs. With emerging Internet technologies such as Java, other down-loadable protocols can be used between different types of Web clients and servers. These protocols can be either stateless or persistent, whichever is best suited to application-specific requirements. They can bypass the underlying HTTP, or they can be built on top of the HTTP protocol or coexist with the native HTTP protocol. For example, JDBC [100] is a proposed open database connectivity standard that can be used to build database-specific protocols (possibly persistent) between a Web client and a database server.

Web clients and servers have common features found in certain distributed computing environments:

1. Coarse-grained. In client-server architecture, due to the computation constraint on client machines it is necessary that only one or a few client programs run on a local client machine. Distributed client programs are in coarse grain. On the other hand, Web servers are usually the gateway to a centralized resource and must be able to handle a large number of concurrent client requests; thus they are also coarse-grained.

2. Distributed and heterogeneous. Web clients and servers can be on different platforms and operating systems, from personal desktop computers to SMP

multiprocessor machines.

3. Message passing. A request for service from a client to a server generally involves the transmission of messages in each direction to transfer the request to the server and to return the reply to the client. Until the messages have been transmitted, the client must wait. Thus the rate of progress in the client’s work depends upon the speed and responsiveness of the network. Figure 5.1 shows in simple form the sequence of actions by the client, the network, and the server when a request is processed.

As shown in Figure 5.1, we can view a simple Web client-server interaction via HTTP as a special form of the dataflow model described in Chapter 2. Essentially, the client-server model based on HTTP is a dataflow model characterized by two key features:

1. Activation of a process is solely dependent on the availability of the arriving input data (client request or server result).

2. Message passing between modules (processes) is only via predefined input/output ports. We can view a typical Web client or server as having only one input port and one output port in the dataflow pipeline. Usually, the port number in a URL defines such an input port on the server, while the client uses a single port for both input and output.

5.2 WWW and Java: A New Paradigm Shift in

Distributed Computing and Software Engineer-

ing

The emerging Web and Java technology extends the base capability and functionality described in the previous chapters and opens a spectrum of opportunities for metacomputing software development in areas that include:

CHAPTER 5. WWW AND DISTRIBUTED COMPUTING 86 Figure 5.2: The Web client-server model as a special case of the dataflow integration model 1. WWW virtual machine as a world-wide meta-computer for informa-

tion processing/distribution and parallel/distributed computing. It is generally agreed that the Web will become a new software environment and an innovative operating system interface that will support both compute-intensive and data-intensive distributed applications in which a Web browser provides a uniform and portable GUI to access integrated information services and computing resources available through generic Web servers.

2. A Java-based object-oriented software engineering environment for Web meta-computing. Java provides a new programming environment that can well support a dataflow-based software integration framework for parallel and distributed computing. The advent of Java programming moves the conventional client-server WWW architecture towards a server-server architecture and opens new opportunities for WWW distributed computing. The main breakthrough comes with the truly neutral, portable, and downloadable Java programming platform. For instance, a downloaded Java applet can turn a Web client at runtime into a Java server to communicate with other Web servers by using whatever message-passing protocols the servers agree on.

3. New Java-based parallel programming languages. The high-performance Java language or language interfaces from existing parallel programming systems to Java can be developed to bring a new dimension to parallel programming. The object-oriented data parallel language, HPJava [106], is an example.

4. Module authoring, publishing and distributing standards and systems for re-usable module repository, composition, sharing and distribution.

5. Generic and domain-specific WWW-based PSEs. The Web provides all the necessary components and technologies to build distributed PSEs. Furthermore, a PSE provides an ideal application platform with which to bridge the pervasive Web technology and the pyramidic HPCC technology and applications.

Although the current major use of the WWW is focused on distributed information authoring, publication, and delivery, we view the WWW as a more general software engineering environment that can provide a better integration framework to support distributed computing for a broader range of both data-intensive and compute-intensive HPDC applications. Our vision and concept are based on three main observations and findings:

1. The universal and portable GUI provided by a Web browser. The Web browser, for example, Netscape Navigator from Netscape, Inc., or the Internet Explorer from MicroSoft has become a desktop windowing front-end that is independent of platforms and operating systems. It creates a Web-window environment that provides a visual and topmost interface with which to access various local or distributed computing services. All distributed resources are transparent to an end-user under this interface. It presents a “virtual world” picture on a local machine in which components actually come from geographically distributed places. For our proposed framework, a Web client can be the GUI front-end of a PSE to provide the following functionalities:

(a) Visual programming interface for composing and configuring flow networks from distributed reusable modules. Related work can be found in [89].

(b) GUI for capturing user input and displaying output.

Instead of developing new software, we can reuse most of the Web client technologies for the component technologies required by a particular PSE. The emerging Java programming provides an excellent client programming environment for GUI design, networking, and graphics. Web-Windows [114] explains this concept in more detail and its implications for HPCC and MPP programming environments.

2. “Compute-Web” servers as distributed daemons in a general-purpose distributed computing system. The WWW itself represents the largest meta-computer, with some 20 million potential nodes world wide. This potential is expected to grow by a factor of 10 as the Information Superhighway is fully deployed. By extending a Web server’s computational capability and using HTTP protocol for point-to-point message passing between distributed Web servers, a naturally integrated distributed computing environment in the WWW can be built on the sessionless and dataflow-based client-server framework. For example, a simple proof-of-concept, Web-based distributed computing framework based on the extended “Compute-Web” servers using the CGI extension has been proposed in [109]. With emerging and fast-growing Web technology such as Java, there can be many approaches for extending current WWW client-server architecture into a fully distributed computing environment. Instead of exploring different approaches toward building such a Web-computing framework, we emphasize here the concept and feasibility that Web provides for a general software integration environment for networking applications. In Chapter 6 we outline an implementation plan for the proposed dataflow framework in WWW and Java.

3. The World-Wide Web Virtual Machine (WWWVM). The WWWVM is the distributed software basis for a publication-based model of computation in which existing Web servers collaborate to provide simple, coarse-grained dataflow among Web-published software modules. Dataflow is the fundamental message-passing paradigm for WWW-based distributed computing. The core technologies of WWWVM are the existing standards for client-server interaction and data exchange now in common use (CGI, HTTP, HTML, MIME).

Clearly, the current Web is not the place in which to explore complex parallel algorithms with stringent latency and synchronization requirements. However, the Web provides an excellent integration framework for coupling certain HPDC applications at the task-parallel level that require only “embarrassing parallelism” and that are suitable for the high functionality, but modest performance, of the Web. It naturally inherits a coarse-grained software integration model and is insensitive to the modest bandwidth and high latency of current geographically distributed HTTPD servers. Two major areas in which we believe the WWW can play an innovative role in linking various parallel and distributed computing components are Web/RDBMS and Web-based collaborative systems.

5.3 Web and RDBMS Integration

Over the last twenty years many new technologies have been invented in both networking and database areas, with each driven by and focusing on its own application needs. The representative achievement in networking is the Internet as a major vehicle boosting the networking industry, while in the database world the maturation of the relational database management system (RDBMS) enjoys the most widely accepted commercial application among computing machinery. Although both worlds adopted certain techniques invented in the other area, such as networking techniques of the client-server, distributed database, replication found in database systems, and common database techniques of data storage, caching, indexing and searching in networking applications, their development has largely been done separately. The emerging WWW applications on the Internet provide both opportunity and incentive to integrate them in order to build interesting NII applications.

A typical Web server has a data repository to be accessed upon Web client requests. This data repository is structured in HTML pages that correspond to ASCII files stored in a file system on the Web server host. In terms of information management, the current data repository approach used in the WWW suffers the following limitations:

1. Static HTML pages. It is difficult to keep HTML pages updated automatically. HTML files are unstructured and full-text oriented. Modifications of HTML pages are usually done manually. No data consistency can be validated.

2. No application logic in data storage. Hyper-links are the only means with which to represent logical relationships among information entities, which is only a hierarchical relationship. Application logic has either to be built into sophisticated HTML page design or simplified. No data integrity concept can be enforced.

3. A plain file system for data management. The file system usually provides only primitive functions such as open/close, read/write operations to raw data files.

A database management system provides a software system developed to store, manipulate and manage data in a reliable and efficient way. By integrating an RDBMS server into the WWW, a Web server can be enhanced with:

1. A dynamic data processing engine. All client requests are assembled as data queries to the database engine for real-time processing, and query results are generated on the fly so that they are the most updated and consistent with other data entities in the database. Data caching and access concurrency control are handled transparently by the engine, making it possible for access and query performance to be optimized with well-established database optimization techniques such as shared query-parsing, intraparallelism, and cost-based query optimizations.

2. Conceptual representation and organization of data entities and their relationships close to the real-world. Information is no longer stored as raw format in physical data files. Instead, it is stored in tables that fit into an application logic and conceptual structure of the application domain. An RDBMS server employs a simple, yet powerful, relational data model for information management and processing.

3. A backend search engine supporting complex queries. Other than simple file access, an RDBMS can supply various views and data access patterns to the same data in the database, using the well-established Structured Query Language (SQL) for sophisticated database programming and HTML page customizations.

On the other hand, the WWW brings a new dimension into traditional database applications in that:

1. The WWW provides a portable and transparent data access and networking interface to centralized remote databases, independent of client platforms and operating systems. HTTP can be the data transport protocol between a generic Web/DB client and an integrated Web/DB server. No proprietary client-server protocol is needed for distributed data access, and a truly portable networking protocol can be standardized to facilitate data access of a single portable client program to multiple, distributed databases from different vendors. This has been a long-sought goal by ODBC in database industry.

2. The WWW browser can be a universal GUI for hypertext-based data retrieval. It can be a common software environment in which various database client programs can be embedded and enriched to integrate with other software components required in a database application. For example, the search interface for ordinary end-users can have the same Web-based look and feel as that for a system administration used by a database administrator.

3. Together with the advances in Internet networking infrastructure, the WWW provides an opportunity to build a truly globally-shared information system for maximum data dissemination and sharing. Valuable information stored in a central place can easily be accessed from anywhere around the world.

4. With the rich presentation and publishing advantages in the WWW, an integrated multi-media information system can be built for data-intensive applications by combining other emerging WWW technologies with RDBMS systems.

The open architecture of a client-server model serves the common base to naturally link RDBMS servers into the WWW environment. Figure 5.3 gives the configuration CHAPTER 5. WWW AND DISTRIBUTED COMPUTING 93

Figure 5.3: Linkage of Oracle RDBMS servers to the Web of an integrated Web/Oracle system. It is used in this chapter’s case studies. More system integration issues will be covered in the case studies.

Parallel RDBMS technology represents a major maturation and adoption of the general-purpose parallel computing and software in the commercial world. Targeting data-intensive applications that require a performance of data-processing beyond that of a single RDBMS server, a parallel RDBMS employs a wide range of parallel processing approaches on general-purpose distributed and parallel systems, including parallel I/O or parallel file systems, parallel query algorithms, high-speed networking, data and task decomposition, and parallel cache management. Appendix D.1 gives a technical overview of the parallel Oracle7 RDBMS server on an IBM SP2.

In the next case study we point out that by integrating the WWW with parallel RDBMS technology, such as an Oracle parallel server on an IBM SP2, we can achieve the seamless deployment of high-performance server technology into the reach of a wide range of users. The user and application base is far bigger than the traditional HPCC community, as WWW software has rapidly become a popular software commodity with fast-growing acceptance in society, industry, and large installations of Web-client and server software around the world. We expect the open WWW architecture and rich environment to attract more and more of both traditional and new applications and that distributed computing will eventually shift to a new paradigm of Web-top software engineering environment.

5.4 Case Study I — A Web and Parallel RDBMS-

Based Information System for Internet Collab-

oration

This section describes how a parallel Oracle RDBMS server is integrated into the World Wide Web environment to build a powerful Internet-based information system for a set of email-based collaborative applications, including a USENET newsgroups archive, a mailing list archive, searchable mailbox, and a chat/bulletin board. Major development issues in system architecture and interface design are highlighted.

By integrating Oracle as a backend search and computing engine into a WWW server to store and manage email-based information entities, we build a Web-based collaborative information system. This system can facilitate and archive Internet email-oriented collaboration in the forms of USENET newsgroup articles, mailing lists, personal mailbox emails, and chat and bulletin board messages. It has a hypertext-based search interface that can access a variety of search capabilities built on the structured and nonstructured attributes of email messages, including search by keyword, subject, sender, date, thread, and URL.

This case study illustrates the concept of bringing traditional database technology into the Internet and a Web environment so that the WWW becomes an integral part of data access and generic interface to a parallel database system. Most important, an RDBMS becomes a powerful computing backend and information management engine to a WWW information space.

The prototype system is configured on an MIMD parallel machine IBM SP2. The high-performance server technology of parallel Web servers and parallel Oracle RDBMS servers is employed to demonstrate the integration of parallel architecture into WWW applications. Appendix D.2 describes in detail this integration work.

5.5 Case Study II — WWW Search Systems Us-

ing SQL*TextRetrieval and Parallel Server for

Unstructured Data

The WWW can be viewed as a giant distributed database (or data repository) where data entities are stored as HTML pages on Web servers that are mostly nonstructured or poorly structured text documents and where logical relationships among Web pages are represented by “hyperlinks” in HTML. The interactions between a Web browser and (remote) Web servers via HTTP can be seen as the (network-based) access interface/transactions between the database client and server, where browsing, i.e., clicking by following hyperlinks, is the simplest and most primitive search/query activity in the database such that no sophisticated search engine is required except for a file system on the Web server. Depending on the client’s browsing scope, the content of the distributed database can be as small as a single Web site, or it can be as big as all the Web servers on the whole Internet. With the daily growing volume in WWW space, it becomes impossible to rely solely on “browsing” to quickly and accurately locate/discover relevant information on the WWW. One natural solution is to build Web search systems. For the distributed WWW database, one can view such a Web search system as a centralized meta-index database where text indexes of the distributed data repository are built and stored at a central place to speed up the “browsing” process.

There have been many approaches for building such a Web search system. For example, Yahoo [85] used a “Yellow Page” paradigm to catalog Web servers by some predefined criteria. While many WWW indexing services like Yahoo adopt such approaches close to WWW technology, we are primarily interested in employing a well-established, mature database and information retrieval technology to build Web search systems. This approach has also been used in current main-stream Web search systems, including Alta Vista [81], OpenText [84], Lycos [83], and InfoSeek [82].

In this case study we describe our experience in developing Web search systems using Oracle’s SQL*TextRetrieval. In the prototype system we index/store HTML documents of remote Web servers, and SQL*TextRetrieval is used to index full text and other structured data in the “Web space” and to provide an efficient search engine for free-text search. The Web enables global access to and maximum information sharing with a hypertext-based text retrieval system. Using Oracle’s free-text retrieval technology, various search options are implemented, including basic word stemming and phrase, fuzzy, and soundex searching, as well as more advanced proximity search and concept search. The prototype system is implemented on a multiprocessor parallel machine IBM SP2 where a parallel Oracle 7 with parallel server and query options is used. This case study shows how high-performance parallel server technology (a parallel RDBMS server on an MIMD parallel system) can be integrated into the WWW to build advanced information systems with which to deal with unstructured data elements. The prototype system is a fully integrated Web-search system consisting of a Web gathering system, an indexing system, an RDBMS-based search engine, and a Web-based search interface.

A detailed description of this case study and the prototype system can be found in Appendix D.3.

5.6 On-going Research and Development in WWW

Software Engineering

Many similar investigations and development efforts along the theme of this thesis have been conducted recently in the area of emerging WWW and object-oriented technologies. We expect more in-depth work with significant results to be achieved in the following research issues and related areas:

1. Extending object-oriented technology to a software component architecture and system. Work in progress includes CORBA [120], the emerging WWW and Java technology (in particular JavaVM [105] and JavaBean [99]), WWW/CORBA [121] and WWWVM [90].

2. Software libraries and runtime systems for creating distributed applications from collections of reusable software modules. More specific research topics include graphical user interfaces for specifying topologies of distributed applications and direct manipulating state and connectivity of remote modules, a graphical coordination language to separate communication from computation for a uniform communication abstraction that is independent of languages and operating systems, and visual programming interfaces and scripting languages. Examples: EUPHORIA [118], Java Workshop [111], ActiveX [119], WebFlow [89], [115], AVSExpress [1], and Darwin [97].

3. Applications that explore the idea of dataflow-based integration architecture to assemble reusable parallel/distributed modules and functional components in HPDC environment. Some of the major efforts in HPCC include MOPAC/AVS, MetaWeb [113], PSEs.

4. Prototype implementation of a component framework in Java, WWW and CORBA. For example, InfoSphere [116], WebVM [109], JavaBean [99], Java RMI [110] and JavaSpace.

5. Extensions of Web servers to include mechanisms/support for WWW distributed computing. Together with the rapid adoption of the Java programming platform, the development of the Java-based generic Web server has been a major effort in a new generation of WWW technology. Major work in this direction includes JigSaw [102], Servlets [104], Jeeves [103], HTTP-NG [123], and FastCGI [122].

6. APIs for specifying various programming interfaces and software environments under an software integration framework. Examples: Java APIs [105], JDBC [100], Java IDL [112], CORBA.

7. Application of a software integration system in PSEs and collaborative systems. Examples: InfoSphere, NexusJava [117], WebSpace [108], MetaWeb, Habanero [107] and WebPSE.

A Meta-Computing Software

Development Environment

6.1 Software Integration Approach in Meta-Computing

The complex problem structures and domain found in a meta-application require that the programming paradigms/models used to represent problem structures also be complex and heterogeneous. Mixed programming languages and paradigms are expected to be integrated in order to collectively solve a complex meta-problem. Problem abstraction and task decomposition requirements in a meta-problem imposes the following requirements on software engineering:

1. Abstraction of task/module and data objects in an integration framework

2. Templates, modules, and objects concepts at language level and framework level

3. Wrappers for existing programs probably written in different languages

4. Problem decomposition essential in abstraction to partition meta-problem into logical, relatively independent smaller pieces

5. Exploit parallelisms at all levels of abstractions

To be able to efficiently and effectively map a complex meta-application onto a meta-computer, a general software integration environment will have the following common features:

1. A single unified integration framework: Components/tools in the environment should be tightly coupled and readily accessible to developers and end-users. This unified framework should be able to coordinate interactions among groups of tools and program components in a flexible way.

2. An abstraction hierarchy of program components and data objects, and their interfaces, should be preserved to allow the refinement of data and task decomposition and implementation.

3. Multiple, yet different, programming paradigms/models should be able to coexist in the framework so that parallelisms at different levels can be exploited.

4. Simplicity and effectiveness: New and existing program components can be easily “glued” together.

5. Extensibility, flexibility, and portability: The framework should allow seamless extension of a meta-problem and a meta-computer; for example, new components can be added into an existing one.

A software integration framework will build a data and execution model that is based on a dataflow message-passing paradigm to permit fine-grained parallelisms within modules, while at the topmost level dataflow execution is used to integrate all functional and software components (program modules) in a HPDC application. The framework will be using restricted data-driven message passing while providing some of the benefits of data-parallel languages such as automatic data decomposition and collective communication and will still allow users to write in a familiar language, using standard software packages.

This process-based distributed integration framework, although originally designed for workstation operation (AVS, SGI Explorer, IBM Data Explore, Khorus, etc.), offers a means by which distributed systems may be connected by workstations or high-performance computing systems. They are characterized by a user-built framework of interconnected modules representing the dataflow for a given application. Such systems support rapid prototyping, plug-in-like modular composition, interactive application steering, and location independence. From the application writer’s perspective, this approach relieves the programmer of needing a detailed understanding of architectural details and offers significant software savings through the reuse of modules. Because the interfaces between modules are well defined, modules developed at different computer sites/platforms may be freely shared.

The dataflow network constructed at runtime supports distributed computing in a heterogeneous LAN or WAN network environment of high-performance systems and graphics workstations. Modules themselves perform the underlying work of data input, simulation computation, fine-grained parallel programming in data-parallel or message-passing computation, data mapping, data transformation, rendering, and output. It hides the inherent complexity of different internal numerical representations, byte ordering, vector or scalar visualization algorithm dependencies, or serial or parallel I/O in a modular HPDC heterogeneous framework. Overall, it gives the user control over the optimization of heterogeneous resources and network capabilities.

Users may interactively steer simulations through the adjustment of an application’s critical parameters. In a similar manner, modification and adjustment of the process itself may be performed in an interactive manner. The same framework that facilitates the easy creation of complex application tasks also provides for its quick modification to include new and different data-mapping and parallel-processing functions.

The modular approach of the framework lends itself well to software sharing, module reuse, extensibility and flexibility. Software sharing means that modules created for one high-performance architecture can readily be ported to another similar architecture. The reusability aspect of the system means the HPDC parallel code and algorithm are modularly developed and compiled once, then reused indefinitely in a plug-and-play mode in a variety of diverse applications. The extensibility of the framework is such that users may write their own modules in their favorite HPDC language/program binding. System adaptability is a key goal in software engineering and remains a major feature in integration architectures. Custom integration approaches usually yield point-to-point integration solutions, and the system becomes difficult to extend and modify due to the complexity and hardwired solution for coupling components. This dataflow-based approach provides an open interface and flexible architecture to realize adaptability on a meta-computer for HPDC applications.

6.2 A Layered Meta-Computing Software Engineer-

ing Environment

A meta-computing programming system should have a layered software engineering environment to hide system complexity from end-users and to represent problem and program abstraction hierarchy. As proposed in [128] and shown in Figure 6.1, a meta-computing software system has in general three major layers:

1. Application specification layer. End-users and application developers use this layer to develop and configure a meta-application in a conceptual and modular fashion. Using a coarse-grained, task-level, functional decomposition mechanism, a meta-application is represented as a task graph in which each task is independent of the programming model, programming language, and computer architecture. A visual programming interface is essential to allow developers to describe task decomposition and their relationships graphically. Another important component in this layer is a high-level description language to represent task descriptions of a meta-application abstraction, independent of its metacomputer implementation. In the JavaFlow framework, to be described in the next section, we develop a Network Editor to create, connect, and formulate a dataflow-based task graph from a “drag-plug” GUI interface. A FlowScript language is used to record the application specification and task graph configuration.

2. Application configuration layer. Mapping from a task graph to a virtual meta-computer is conducted at this layer. Given a library of task implementation and a description of a physical meta-computer, a task configuration manager allocates resources to map/bind task programs onto a physical metacomputer configuration. An optimized scheduling mechanism can be considered to achieve high-performance program execution and load balance. The result of this layer is a task-node configuration to be scheduled to run on a meta-computer infrastructure. In JavaFlow, we propose a virtual machine concept of JavaFlow VM that is used to separate the construction of task and meta-computer graphs from physcial resource allocation and task-node mapping.

3. Application execution layer. This layer operates as a runtime system of this meta-computing software environment. It schedules the execution of the allocated tasks on a physical meta-computer configuration. In JavaFlow we use a generic Web/Java server technology “Servlet” to deploy dynamically the dataflow execution runtime system on a configuration of WWW servers that consists of the meta-computer software/hardware infrastructure.

The next section elaborates on key components of a specific layered meta-computing software engineering environment.

6.3 JavaFlow — A Java Implementation Frame-

work of the Dataflow Model on WWW

In this section we outline JavaFlow, an implementation plan of a software integration system to support publishing, composing, and executing distributed Java programs/modules for distributed computation. Javaflow is a design realization of the proposed general dataflow framework in Chapter 2 on a Java/WWW platform. Major architectural and design issues in building the proposed framework are touched upon. The implementation is based on an object-oriented system that supports messages Chapter 6. A Meta-computing Software Development Environment 104 Figure 6.1: A Three-Layered Meta-Computing Software Development Environment and threads. It uses a data-driven execution model to coordinate parallel and distributed coarse-grained Java and other software component programs in a Web/Java programming environment.

As shown in Figure 6.2, major components and key system issues in Javaflow include:

1. A JavaFlow Virtual Machine (JavaFlow VM)

2. JavaFlow servers

3. JavaFlow modules

4. A JavaFlow protocol for dataflow message-passing

5. JavaFlow API

6. A JavaFlow runtime support facility

7. A JavaFlow module and VM node registration subsystem

6.3.1 A JavaFlow Virtual Machine (JavaFlow VM)

A JavaFlow VM is a meta-computer for coarse-grained task parallel computing and software integration in a dataflow message-passing paradigm. It consists of a directed graph of JavaFlow nodes where nodes represent virtual processors, each of which runs a JavaFlow server, and edges represent dataflow directions and dependencies between connected nodes.

Execution of a JavaFlow application involves three steps:

1. Construct a JavaFlow module network that represents a task-parallel application. A JavaFlow module repository and a JavaFlow editor are designed to allow module composition from a library of system and user-defined, Chapter 6. A Meta-computing Software Development Environment 106

Figure 6.2: The JavaFlow Architecture and Major Components reusable JavaFlow modules through a “drag-plug” visual programming interface. A JavaFlow module registration subsystem is designed to provide a standard mechanism with which distributed developers can author, publish, and contribute user-defined JavaFlow modules to the module repository.

2. Construct a JavaFlow VM so that the module network in (1) can be effectively mapped to the JavaFlow VM. Because a JavaFlow VM is a virtual machine and a JavaFlow network uses a directed acyclic graph as network topology, theoretical or heuristic algorithms can be developed so that an optimal mapping can be achieved by using resource and performance metadata gathered from a JavaFlow VM node registration subsystem that is designed to register, store, and share JavaFlow servers among willing participants.

3. Allocate a JavaFlow VM runtime configuration. Mapping from the JavaFlow VM to physical processors includes the allocation of processing resources, setup of execution facility, and deployment of the module network. A JavaFlow Loader is designed for this purpose.

The JavaFlow VM runtime configuration can be a loosely distributed system consisting of machine hosts over a WAN (such as the WWWVM) or a network of workstations over a LAN (such as a cluster), or a tightly coupled MIMD or SMP parallel system. The only system constraint is that each JavaFlow VM node must support JavaVM [105] runtime and that the network supports the TCP/IP layer.

Note that when a JavaFlow VM is mapped to a WWWVM (i.e., Web servers as VM nodes), a one-to-one mapping is expected, as a JavaFlow server is essentially an HTTP/Java server with extended facility, although one or more Web servers can be mapped (hosted) on a single physical machine.

The concept of a JavaFlow network, a JavaFlow VM, and a VM runtime configuration and their mappings gives three system design advantages:

1. Construction of a JavaFlow network can be completely application-oriented, independent of choices of physical processors and network topologies.

2. Allocation of physical resources can be delayed until execution, independent of module composition and the mapping from a module network to a JavaFlow VM. An optimized allocation scheme can be sought at runtime and a faulttolerant configuration can be supported. For example, Figure 6.2 shows a 5node JavaFlow VM configuration. The only allocation dependency is the local client machine V and the remote host D running native Java class interfaces to non-JavaVM systems. If nodes I, II, and IV all fail before or after the initial allocation, the architecture is able to support a dynamic remapping and reallocation scheme using a 2-node VM configuration consisting of nodes V and III. Modules A_{1}, A_{2}, B_{1}, D_{1} and D_{2} are all reallocated to node III.

6.3.2 JavaFlow Servers

A JavaFlow server consists of an HTTP server, a JavaFlow Daemon, and one or more JavaFlow application modules. A special JavaFlow server is on a local client machine where a Java-enabled Web browser replaces the HTTP server to load a JavaFlow Daemon and modules as applets. A JavaFlow VM runtime system consists of a set of JavaFlow servers with loaded JavaFLow Daemons and modules.

The HTTP server is written in Java in order to be able to support the standard Java servlet [104] mechanism for spawning external persistent processes — for example, Jeeves [103] and JigSaw [102]. The main functions of the HTTP server in a JavaFlow system include:

1. Provides a JavaVM runtime and programming platform to execute JavaFlow modules.

2. Using HTTP, it loads the JavaFlow Daemon and module servlets over the network or from local disk at runtime.

3. If required, it uses HTTP to access conventional WWW information and resources on the remote host.

Once the daemon and modules are loaded on a host, the HTTPD server will no longer participate in further JavaFlow execution and is thus free to serve other standard WWW services. The JavaFlow control thread is then transferred to the daemon for JavaFlow execution.

In the data-driven computation model a JavaFlow module can bind one of its output ports to an arbitrary number of input ports. Methods invoked on an output port are broadcast to each input port to which it is bound. Messages from an output port to an input port are delivered in the order sent. The message stream delivered to an input port is a merge of message streams sent to the input port to which it is bound. The only means of interaction between JavaFlow modules is message passing sent through dynamically connected JavaFlow channels between output ports of a module and input ports of another. A module doesn’t share an address space with other modules. It has a persistent state and must be executed whenever its input ports change. The underlying data transfer and synchronization is actually carried out by participating JavaFlow daemons. A JavaFlow daemon is a distributed flow execution manager that implements the daemon-to-daemon and daemon-to-modules message passing in JavaFlow protocol and ensures dataflow execution and data transfer. Major tasks of a JavaFlow daemon include:

1. Accepts incoming messages for input ports of all modules. For example, at startup a separate thread should be created to listen in a loop to a particular port for incoming message streams and to dispatch them to corresponding handlers for message deserialization and buffering/queuing.

2. Delivers outgoing messages for output ports of all modules. For example, using a JavaFlow protocol, at the request of a running module a thread will be created to get a serialized Java object stream and then broadcast it to all the daemons whose JavaFlow modules have an input port bound to the output port.

3. Synchronizes the asynchronous incoming/outgoing data stream requests and module executions, including channel/module registration, message queuing and buffering, module activation/suspension, state monitoring/persistence, resource sharing, error handling, etc.

For data transfer between two JavaFlow daemons on different hosts, a JavaFlow data transport protocol is used. For data and control flows between a daemon and a JavaFlow module or between two JavaFlow modules on the same host, the industry standard protocol JavaBean [99] can be used. Note that data transfer between two JavaFlow modules on the same host must be supervised by the daemon. Data transfer between two JavaFlow modules on different hosts is coordinated through the two onbehalf daemons.

JavaFlow modules in a JavaFlow server are Java servlets or applets loaded for individual application tasks.

6.3.3 JavaFlow Modules

The fundamental unit of computation in JavaFlow is the module. Modules which process inputs to generate outputs are intended to be fairly coarse-grained units of computation. A JavaFlow module is a stand-alone Java applet or servlet that consists of two distinct parts, the kernel and the interface. The kernel is a typical Java program body in JDK or various Java APIs for task and object-oriented computations. The JavaFlow interface defines dataflow input and output ports associated with each JavaFlow module, using a JavaFlow API. A JavaFlow module is actually a wrapped Java applet/servlet in JavaFlow API. Conversion from a standard Java program to a JavaFlow module should be straightforward and simple so that automatic generators may be developed to simplify JavaFlow module authoring and wrap-up.

There are generally two categories of JavaFlow module: system and user-defined. System modules are JavaFlow system building blocks such as the JavaFlow daemon, JavaFlow Editor, JavaFlow Loader, JavaFlow Registrar, or general-purpose modules such as RDBMS readers, common GUI instruments, data filters or compressors, 2D image viewers, 3D VRML viewers, audio/video players, Web agents, etc.

A JavaFlow module can be either downloadable or native. A downloadable JavaFlow module is completely written in JDK and Java APIs with imported JavaFlow classes. It is compiled by a standard Java compiler into a stand-alone Java byte-code, to be able to “write once; run anywhere; reuse everywhere,” i.e, downloadable to any remote JavaFlow server or Web browser via the servlet/applet mechanism. A native module is a wrapped JavaFlow module consisting of a JavaFlow part in JavaFlow API and a native part. The native part is usually written in other parallel systems such as MPI, PVM, or HPF to interface with non-Java, fine-grained node programs that are attached to a particular host. For example, in Figure 6.2 the module C_{1}, which provides a Java interface to non-Java systems such as CORBA or PVM, is a native JavaFlow module that must run only on host III. Note that many Java APIs and standards are currently being developed in industry for replacing or interfacing legacy systems. We expect that the majority of JavaFlow modules are downloadable programs. For example, in Figure 6.2 a downloadable module A_{1} may use a portable Java interface JDBC [100] to access multi-vendor legacy databases.

A JavaFlow module can be either a servlet to be loaded to an HTTPD/Java server or an applet to be downloaded by a Web browser. Servlets are similar to applets in that they are Java-object bytecode that can be dynamically loaded off the net. Servlets differ from applets in that they are faceless objects. Most applet JavaFlow modules are source or terminal dataflow modules. For example, in Figure 6.2 applet E_{1} can be a GUI to provide input graphical widgets to get input data on the client machine; applet E_{2} may be an image viewer to display application results.

Note that a JavaFlow module is different from a Java object. A module is an active agent whose state can be changed only by the module itself or by a JavaFlow daemon at the request of another JavaFlow module/daemon.

6.3.4 A JavaFlow Protocol for Dataflow Message Passing

As defined in the dataflow execution model, a JavaFlow module interacts with other modules by receiving and sending object data for message passing in a data-driven fashion. The JavaFlow protocol defines when and how object data is transferred between JavaFlow daemons and modules in a reliable manner, and how data transfer and program execution are synchronized to guarantee a deterministic dataflow computation semantics.

While TCP/IP is the base data transport protocol on which JavaFlow protocol is built, there can be many approaches for defining and implementing a data-driven protocol. Instead of giving a detailed protocol specification here, we merely identify several important issues.

Each JavaFlow module has a set of predefined input ports through which it receives input data and a set of output ports through which it sends output data. Each of them has a unique identification that can be communicated between modules. Javaflow modules and their associated ports are the only objects with a global identification scope that is visible to the rest of a JavaFlow application.

A JavaFlow daemon handles messages that arrive and messages to be sent on behalf of running JavaFlow modules. For example, it can queue and buffer incoming/outgoing message streams for each JavaFlow module.

Message passing between a JavaFlow daemon and modules or among JavaFlow modules on the same host should use the most efficient mechanism possible, such as shared memory or message queues. The emerging industry standard component model JavaBean API can serve this purpose. In JavaBean API, a JavaFlow daemon can be implemented as a parent container and JavaFlow modules as components. This provides the context for module components to be arranged and to interact with one another. More specifically, two of the proposed JavaBean services can be used— namely, the persistent service that allows a JavaFlow daemon to buffer incoming and outgoing message streams and to store the state information for active modules, and the event-handling service for events raised in a JavaFlow applet module to be delivered to the JavaFlow daemon and appropriate modules on the same host.

Objects sent by a JavaFlow daemon are usually converted into a string, sent across the network, and then reconstructed into their original types by the receiving server. Java methods are used to convert an object to a string and to create an instance of the sending object at the receiver. A more advanced mechanism, such as Java serialization [101], can be used to create a serialized representation of objects or a graph of objects with sufficient information to insure that the equivalent typed object can be recreated.

The actual implementation of the daemon-daemon protocol has to consider a trade-off between reliability and performance. It can be sessionless, similar to HTTP, to achieve reliable message passing over wide area networks. A sessionless protocol usually has a poorer performance and higher latency than a session-oriented protocol. Due to the coarse-grained feature in both dataflow requests and server/module granularity, an HTTP-like protocol with extended support for Java serialization may be a good solution. The base protocol should be message-based so that the most recent input messages received for a module can be buffered. For stream-based data types such as audio/video feeds, a connection-oriented protocol extension has to be developed and a direct port-to-port transfer channel should be used to feed stream data between two modules, bypassing the buffering mechanism.

In the current architecture we rely on standard HTTP protocol to load JavaFlow modules to distributed JavaFlow servers over the network.

6.3.5 JavaFlow API

JavaFlow API defines a Java programming interface to allow a JavaFlow developer to write, author or wrap a JavaFlow module. While the computation part of a JavaFlow program is usually written in Java Core API or other Java standard APIs such as JDBC and Java Media, JavaFlow API specifies Java data types, object classes, and interfaces that are specific to the JavaFlow system. In the form of the importable Java class library, JavaFlow API should include two major functional components:

1. Primitive and composite data types. Each input and output port has an interface type associated with it; this type defines the types of object data that can be sent through the output port or received from the input port. All classes in Java Core API should be allowed as primitive JavaFlow data types. For composite data types, selected classes defined in major Java APIs can be considered; for example, file and file directory such as HTML page, relational table in JDBC, audio/video stream in Java Media. User-defined data types should also be supported. One of the major advantages in adopting Java programming as a uniform and object-oriented software development platform in that JavaFlow data are object-oriented. This means that when a JavaFlow object is transferred, all operations (methods) defined on the object type are implicitly included. This is significantly better than stream-type systems such as PVM and AVS in which explicit conversion procedures must be used in a module to pack/unpack a data stream. In the JavaFlow system this can be transparently handled by the Java Serialization mechanism.

2. Interfaces and classes to define a JavaFlow module and associated ports. These include initializing and describing Java programs to JavaFlow, filling objects in output ports or fetching objects from input ports, error handling, etc.

6.3.6 A JavaFlow Runtime Support Facility

This facility has three components:

1. A JavaFlow Editor. This provides a visual programming interface in a Java applet for creating, revising and starting a JavaFlow network consisting of JavaFlow modules from a repository of reusable JavaFlow modules. A JavaFlow network represents a distributed application described as a coarse-grained task graph in data-driven execution. A JavaFlow Editor should consider these major issues:

(a) From the client machine this applet accesses to a module repository on one or more Web servers and makes graphical presentations for each module and its ports.

(b) Allows a user to interactively compose a JavaFlow network by drag-plug using a mouse to connect network channels. A channel must agree on the same data type on both ends.

(c) If a user chooses to explicitly bind/map the constructed JavaFlow network to a JavaFlow VM runtime configuration, this interface should also provide visual means to interactively facilitate the mapping by accessing to a JavaFlow server repository on a repository server. Note that certain modules have particular node dependency.

(d) A constructed JavaFlow network, mapped or unmapped to a JavaFlow VM runtime, should be represented in a JavaFlow script language (FlowScript), to be sent to a JavaFlow Loader on the JavaFlow Repository server to start the deployment and execution of the JavaFlow application on a JavaFlow VM runtime system. A user can also save it as a JavaFlow application for future reference.

2. FlowScript, the JavaFlow script language. FlowScript is a readable text language designed to record basic user interaction and a JavaFlow network configuration in the form of script files or streams. The major purpose of the scripting language is to separate and bridge the process of a JavaFlow network construction in a network Editor from the process of mapping the network to a JavaFlow VM runtime system in a Loader. Module networks in FlowScript are interpreted by a Java Loader on a Repository server. We expect the script language to be HTML-like, in particular, the URL form of servlet syntax should be preserved, i.e., a servlet with a source host and a destination host is denoted as http://<server host>:<port>/servlet/<servlet URL>, where the <servlet URL> is a regular URL that points to the location of the servlet. For example, to represent a servlet A_{1} stored on a Web server

http://npac-repository/servlet/A1 that is to be loaded on another Web
server http://greatwall:1963, the URL is
http://greatwall:1963/servlet/http://npac-repository/servlet/A1.

FlowScript is also used to save networks and parameters settings that can later be read for modification or execution with or without requiring the user to use the JavaFlow Editor directly.

3. A JavaFlow Loader. Using a JavaFlow network configuration in FlowScript and a JavaFlow VM node repository, the Loader servlet running on the repository server performs dynamic mapping/allocation from the module network to a JavaFlow VM runtime system. It needs to check at runtime the availability of nodes and networks. Certain optimization techniques may be used to find an efficient mapping. After the runtime configuration is chosen, the Loader drives remote HTTPD servers to load JavaFlow daemons and modules servlets over the network or from local disks via HTTP. Note that the daemon and module applets to be run on the local client machine must be loaded by the Editor.

6.3.7 A JavaFlow Module and VM Node Registration Subsystem

This subsystem is used to build a JavaFlow module repository and a JavaFlow VM node repository. Together with the JavaFlow API, it provides a standard mechanism to:

1. Allow module developers to publish, contribute and share standard user-defined JavaFlow modules to a central or distributed module repository. Benefited from a URL representation of servlets/applets, only a symbolic identification is needed to store a JavaFlow module information in the repository. The actual Java bytecode can be kept on any contributing Web server over the network.

2. Allow individual Web servers to be added into a server repository to form a meta JavaFlow VM. A JavaFlow VM node presence in the repository is also a text form such as in URL.

Note that for a native-class module, both the module and the remote host must participate in the JavaFlow runtime system. Thus the two identification data are stored together.

6.3.8 Summary

A typical procedure for a developer to create and run a JavaFlow application can be:

1. Write JavaFlow modules in Java and JavaFlow API and place them on a Web/Java server that supports servlet extension.

2. Publish the modules by using a JavaFlow Registrar applet on a known JavaFlow repository server. If you are willing to participate in the JavaFlow VM configuration, you may also publish the Web server using the same interface.

3. Use the JavaFlow Editor on the repository server through a Web browser to construct a JavaFlow network consisting of your own or other modules available on the repository server. You may also want to construct a JavaFlow VM and map the network onto it, using the same Editor. Or you can leave the system to find an optimized mapping for you.

4. From the same Editor interface, start to execute the JavaFlow network.

Actually, step 1 and step 2 can be optional. A user can start to build a JavaFlow module network by directly re-using existing modules from one or more module repositories. By providing the JavaFlow Registration mechanism, we expect that many JavaFlow applications can be created in this fashion, free of programming by a JavaFlow user.

In summary, this JavaFlow implementation has the following key features:

1. It is Web-based. Web servers are extended virtual processors for general-purpose distributed computing.

2. As both a uniform programming environment and a software engineering platform, Java becomes the single vehicle to carry out the implementation of the proposed software integration framework. It allows the combination of object architecture, Web architecture, and a component architecture for an integrated programming platform, and a new software engineering environment for a wide range of HPDC applications.

3. This open architecture and objected-oriented approach allows us to adopt the open system in Web and Internet to be able to grow with the emerging evolution of WWW technologies, Java, in particular. For example, JavaBean is a major effort in the Java community in this direction. It is an industry component technology standard to facilitate composition/interaction of reusable Java programs in a seamless, independent, transparent and portable fashion, focusing on the software engineering aspect of Java programming and engineering.

4. This plan is truly portable and independent of platforms and operating systems. It frees developers from having to know details of platforms, machine architectures, and network topologies.

5. This is a fully distributed implementation in that there is no centralized facility to coordinate control and data flow among distributed programs. The protocol implementation can consider the performance and reliability trade-offs of a network that may have medium-bandwidth and high-latency and yet consist of different data-transfer protocols.

6. Utilizing existing and emerging WWW/Java technologies, the overall implementation is built on a tiny kernel with a modular architecture. Most JavaFlow components are stand-alone Java programs or even JavaFlow modules, including the JavaFlow daemons, the JavaFlow Editor, and the JavaFlow Loader.

Conclusion

The focus of this thesis work is on proposing a component software assembly paradigm and integration framework for HPDC applications, identifying and specifying software components that can be integrated/composed to create distributed applications, developing a compositional methodology and architecture for constructing distributed applications from reusable components, and implementing a set of parallel applications that demonstrates the methodology. Our major findings are:

1. Overall, a dataflow model can be a feasible and effective integration framework with which to combine data-parallel programming and coarse-grained task parallelism for general-purpose parallel computation. As a restricted messagepassing system, a dataflow execution paradigm allows a flexible and open component architecture that provides a good solution to problems in parallel software engineering, including rapid prototyping, modularity and reusability of fine-grained parallel programs, portability, encapsulation of implementation details, and allocation of resources. Using the execution semantics inherent in dataflow computation, high-level programming constructs such as messagepassing ports and channels and processors allocation/mapping can easily be supported and their implementations handled by runtime of the integration framework. Thus they are independent and transparent to developers. A dataflow model also provides the opportunity and flexibility to build a visual programming interface in the framework to dynamically construct and compose distributed task modules for a particular application from a repository of reusable modules.

2. The motivation of a dataflow framework in a parallel software environment is encouraged by the successful use and deployment of a dataflow framework in graphics software systems and applications. For HPDC applications such as interactive distributed computing and visualization applications that require both parallel computation and data visualization, this framework provides a common ground with which to seamlessly assemble the two components. With its visual programming interface, modular program structure, dataflow-based execution, interactive visualization functionality, and its open system characteristics, a process-based integration framework like AVS provides an excellent framework with which to facilitate integration of various system components into a large, multi-disciplinary application, including the integration of sophisticated interactive visualization, database management, heterogeneous networking, massively parallel processing, and real-time decision making.

3. The coarse-grained model is well suited for combining parallel database systems in distributed data-intensive applications, because the parallelization of database applications is usually obtained by the partition of data placement on parallel disk devices, and SQL-based programs for a parallel and a sequential database server are identical.

4. Support of a heterogeneous distributed environment is one of the major advantages of the process-based software integration framework. Using TCP/IP as the underlying data transport protocol, our work indicates that the proposed framework can support diverse parallel platforms and hybrid parallel computation models. For example, coupling of fine-grained data-parallel and message-passing programs is seamlessly supported in the framework.

5. Choice of optimal platform(s) for particular application needs is supported at the software environment level, including optimization of resource allocation, hybrid parallel computation models, and data redistribution among different platforms and programming models.

6. The coarse-grained framework is an open component architecture that is best suited for composing meta-applications on meta-computers at the scale of a local area network. It can also be applied to distributed multi-media applications over a wide area network. For example, the dataflow execution model can be applied to the World Wide Web meta-computer as a development platform for distributed computing and software integration.

Appendix A

NPAC Meta Computer

The NPAC meta-computer is equipped with computational resources with a variety of the most recent HPCC technologies. Although the structure of installed facilities is not geared toward high volume production needs, it offers considerable computing power. The real asset of the meta-computer, however, is diversity of its resources. Collectively, as a whole meta-machine, or separately, with partial configuration, it is able to support almost every HPCC application and provide testbeds for a multitude of computationally intensive projects, including most of the case study projects described in this thesis.

As shown in Figure A.1, the processing nodes of this meta computer can be broadly divided into three categories:

1. Parallel supercomputer

2. Clusters of high-performance workstations

3. Desktop workstations for program development and data visualization

A.1 Parallel Machines

The parallel machines currently installed include a CM5, an iPSC 860, two DECmpp machines, an SP2, and an nCUBE 2. They represent the major current architectural APPENDIX A. NPAC META COMPUTER 123

Figure A.1: NPAC meta-computer - processing nodes trends for MPP platforms. The configuration and utilization data for these machines are as follows:

1. The CM5 is a 32-node model from Thinking Machines Corporation. Every node has 32MB of memory and, besides a Sparc-1 scalar CPU, four vector units. The nodes are connected by TMC proprietary network with quad-tree topology. The peak computational power of our CM5 is estimated at 3 GFLOPS.

2. The iPSC 860 from Intel is a 16-node model with 256 MB of RAM and with hypercube interconnect topology. It features a small-scale concurrent file system.

3. Two DECmpp machines are large systems in their category. Both are 100 models with 8K and 16K processors, respectively. The DECmpp machines are SIMD computers featuring a large number of relatively unsophisticated processors operating in data-parallel fashion. The DECmpp machines are equipped with a high-speed I/O channel and can communicate over the HiPPI interface. Their architecture makes them suitable for regular problems. The aggregate peak computing power of the DECmpp ensemble is 2.5 GFLOPS. They are ideally suited for image processing and data-compression applications. DECmpp platforms offer relatively sophisticated and user-friendly software support.

4. The SP2 machine is the newest HPCC product of the IBM corporation. Unlike the other machines installed in NPAC, the SP2 consists of a relatively low number (8 thin/4 wide) of very powerful nodes. The nodal processor of the SP2 is an IBM RISC/6000 processor running at 62.5 MHz, one of the most powerful processors available. Internal connectivity of the SP2 is provided by both the Ethernet network and by the high-performance switch with crossbar topology (full connectivity). The parallel processing environment of the SP2 supports both distributed (loosely coupled) and parallel (tightly coupled) programming paradigms. Due to the binary compatibility, the SP2 can be easily integrated with IBM workstation clusters. It is a very flexible, general purpose platform with considerable computing power and is able to support a variety of HPDC applications.

5. The nCUBE 2 installed is a 64-node, 2 GB RAM machine with MIMD hypercube architecture. The salient feature of the NPAC machine is its scalable parallel I/O subsystem. The subsystem, working in parallel with the computational processor array, consists of 32 nodes that form its own hypercube, with an SCSI controller and a disk array connected to every node. The current configuration supports 96 disks with total storage capacity of nearly 0.2 TB. The I/O subsystem also supports networking processors and controllers. It represents a state-of-the-art scalable I/O system, probably the most advanced in the industry. It possesses the capability to support large, high-performance data-processing applications.

A.2 Workstation Clusters

The relatively distributed computing elements include two clusters of high-performance workstations: an IBM R/6000 cluster and a DEC Alpha cluster. An IBM heterogeneous cluster consists of eight workstations (models 550 and 340). Four of the workstations are connected via a high bandwidth, low-latency Allnode switch. Both SP2 and cluster nodes will run the same Parallel Programming Environment software and Load Leveler scheduler. This integration will almost double the CPU power of SP2, presenting users with a single-system view of both the SP2 and the cluster.

The DEC Alpha cluster consists of eight Alpha model 400 compute servers. These workstations represent the newest and most advanced development in workstation technology. The cluster is supported by a high-performance networking backbone consisting of dedicated, switched FDDI segments. This solution provides full FDDI bandwidth and low-latency switching to every workstation in the cluster. The model implemented in NPAC exemplifies Digital’s latest approach to distributed computing.

Together with the DECmpp machines, the cluster will be used to demonstrate the viability of using a highly heterogeneous computing environment to solve computational problems with very irregular structures.

Both clusters of workstations provide meta applications with a unique distributed processing environment. The clusters support a growing class of computational problems that are both irregular (hence difficult to parallelize) and non-vectorizable (hence running inefficiently on vector supercomputers).

A.3 Desktop Workstations

While most of the highly heterogeneous network of desktop workstations are used by NPAC researchers for everyday operation, including program development, document processing, multimedia, and scientific visualization, some of them are also used as part of the meta-machine in a variety of applications. The network includes a variety of workstations from vendors that include SGI, DEC, and Sun. Scientific visualization capability is provided by few high-end SGI workstations and by the IBM graphic workstations integrated with the IBM SP2/workstations cluster.

Overall, the meta-computer represents a state-of-the-art HPCC center providing its partners with access to the newest technology. The unique capabilities of the NPAC computational infrastructure offer a number of benefits that assist NPAC users in developing novel HPCC applications. These capabilities include:

1. A number of different parallel architectures can be gathered in one place. This allows for extensive experiments and for identification of the most suitable architecture for a particular project.

2. In addition to diverse parallel systems, loosely coupled distributed systems are also supported, thus extending the overall meta-machine capability and allowing for integration of the technologies required to solve extremely irregular problems.

3. The integration capability is supported by an efficient networking backbone. This capability will soon be extended to applications running on geographically remote computers by interfacing NPAC’s internal LAN to the NYNet highspeed network. NYNet will also support applications that require very high bandwidth for data exchange between supercomputers, including support for multimedia and telepresence applications in diverse fields.

4. Both traditional “number crunching” and data processing platforms are available. This makes it possible to test distributed applications that have both intensive data processing and computational components. Both database applications (parallel database server) and I/O intensive applications (parallel I/O subsystems) are supported.

5. The NPAC heterogeneous environment makes considerable computing power available to NPAC partners and users of its facilities.

A.4 Networking

Internally and locally, the processing elements of the meta machine system are interconnected by multi-vendor, multi-bandwidth, and heterogeneous local networks (see Figure A.2). The main networking FDDI backbone is built around the Gigaswitch (Digital), a high-speed, low-latency multiswitch with interfaces to both FDDI and ATM networking technology. A section of the Gigaswitch supports the networking of the Alpha cluster. Other sections of the same device support access to NPAC’s file servers and provide connection to the router supporting Ethernet connectivity. A second networking backbone based on the HiPPI technology backbone interconnects the two DECmpp machines and other machines for a single point-to-point connection and high-bandwidth communication.

Many of the modern applications require a bandwidth much higher than what is currently available. To achieve this goal, a regional alliance has been formed to APPENDIX A. NPAC META COMPUTER 128

Figure A.2: NPAC meta-computer - interconnections (networking) implement the ATM-based high-speed network. The network, NYNet, is being supported by NYNEX, one of the baby-Bell companies, and its participants include Cornell University, Syracuse University, Rome Laboratory, and the Museum of Science and Technology in Syracuse in upstate New York, and Brookhaven National Laboratory, SUNY Stonybrook, Columbia University, Brooklyn Polytechnique, and NYNEX Science and Technology Center in downstate New York. The network offers OC-12 capability. NYNet will offer high-speed access to most of the NPAC resources and will allow for extensive experiments in wide-area distributed computing involving resources of the computing centers in all participating institutions.

The FDDI, ATM, and HiPPI networking technologies must be integrated to ensure internal interoperability and remote access to NPAC resources. This goal is realized by the installation of a Challenge, the powerful Silicon Graphics network server. This unit provides a central-point, high-performance gateway between different networking backbones installed in NPAC and will serve the purpose of a HiPPI switch, a HiPPI–FDDI–ATM gateway, and a direct interface to the second OC-3/OC-12 line of NYNet. This can provide support for geographically distributed applications that require gigabit bandwidth technology.

Appendix B

Case Studies of Multi-Paradigm

Programming in the Software

Integration Framework

This appendix details three application case studies used in Chapter 3 to demonstrate the integration of multiple-programming models in a unified dataflow-based programming framework. The three studies are chosen from atmosphere science, computational electromagnetics, and financial modeling. For each application, we describe a prototype system that illustrates system configuration, software integration, performance, functionality, and other issues. Some background information about the CM5 and PVM are also given to help explain the approaches we adopted when working in a specific context of machine and programming tools.

B.1 Parallel Programming on Connection Machine

CM5

A detailed description of the CM5 architecture and software system can be found in [28] and related CM5 documents from TMC. Here, we briefly outline the existing

Figure B.1: Components of a CM5 system

parallel programming environment on the CM5 for the purpose of our discussion in later sections.

The CM5 system combines the features of SIMD and MIMD designs, integrating them into a single parallel architecture. The unique feature of the CM5 provides alternatives with which to exploit both paradigms in a parallel programming environment.

The CM5 architecture is designed to support data-parallel architecture especially well, which implies relatively synchronous parallel programming featuring a single conceptual thread of control and a global approach to data layout and I/O in which arrays of data are laid out across all processors with all elements computed upon simultaneously by their respective processors. Although the conceptual model itself is completely synchronous, in practice processors may asynchronously execute any operations that are independent of other processors (such as conditional operations on locally stored data). The CM5 data-parallel compilers take full responsibility and advantage of this freedom to exploit MIMD hardware capabilities of the CM5. Currently, the CM5 supports data-parallel Fortran (CMfortran), data-parallel C (C*), and data-parallel Lisp (*Lisp) programming languages.

An explicit message-passing model is extended from the data-parallel model on the CM5 by using a package of macros and runtime primitives called CMMD supporting MIMD style low-level communications operations. Programs that use CMMD typically operate in an SPMD(single program multiple data) style in that each processing node of the CM5 runs an independent copy of a single program and manages its own computations and data layout with communication between nodes handled by calls to CMMD message-passing primitives. The message-passing programs can be written in Fortran 77, CMFortran, C, C++, C* or an assembly language DPEAC.

As shown in Figure B.1, each processing node (PN) is a general-purpose computer that can fetch and interpret its own instruction stream, execute arithmetic and logical instructions, calculate memory address, and perform interprocessor communication. All PNs can perform independent tasks or collaborate on a single problem. A control processor (CP) is similar to a standard high-end workstation acting as a partition manager to communicate with the rest of CM5 system through the Control Network and Data Network. CMOST, a parallel timesharing operating system enhanced from UNIX, runs on the CP to make all CM5 resource allocation and swapping decisions, as well as most system calls for process execution, memory management, I/O and access to local networks. The Control Network provides tightly coupled communication services. Optimized for fast response and low latency, its functions include synchronizing the PNs, broadcasting data to every node, combining a value from every node to produce a single result, and computing certain parallel prefix operations. The Data Network provides loosely coupled communication services. Optimized for high bandwidth, it provides point-to-point data delivery. A data-parallel or CMMD message-passing process running on the CP, plus the nodes, is a single process timesharing the CM5 system with other processes.

The current CM programming environment provides some restricted ways to support the full integration of multi-paradigm programming in a single language, mostly through specialized library routines that encapsulate optimized implementation details of the functions provided. As an example, the CM Fortran Utility Library provides convenient access from CM Fortran to the capabilities of lower-level CM software. The purpose is typically to achieve functionality or performance beyond what is currently available from the compiler. CM Fortran programmers can use the library procedures in situations where one is normally tempted to make explicit calls to lower-level software. The utility procedures take CM Fortran array names and other CM Fortran data objects as arguments so that there is no need to convert CM Fortran objects into data types by lower-level software. The CM Fortran Utility Library provides procedures for inquires, random number generation, dynamic array allocation, interprocessor and local data motion, and parallel I/O[29].

In order to exploit the special high-performance hardware arithmetic accelerator (called vector unit) in each PN of the CM5 system, current CM5 programming software provides some limited ways to mix data-parallel paradigms in a message-passing program. Explicit message-passing CMMD programs can use certain language constructs from CMFortran or C*, in which case the programs use both the microprocessor and the vector units of a PN, with serial arrays stored in the microprocessor and parallel arrays in the VU memory. It can thus take advantage of the VUs for high-speed floating-point computations. Parallel arrays can be passed as arguments to CMMD message-passing primitives in much the same way as serial arrays. Within this integration model, data-parallel operations are confined in a “node level” while the overall program is in a explicit message-passing paradigm.

If adopting this embedded integration approach on a CM5, an ideal integrated program would be as follows: Executed as a single process on the control processor (CP) of the CM5, the main program in Fortran77 (or C) would first act as a host sequential program in the CMMD host/node programming model and start a CMMD node program by enabling and broadcasting some data to the node program, which is written in Fortran77, C, or even CMfortran (node-level data-parallel). Then this host program would invoke another subroutine written in CMFortran (or C*). After the control is back to the main program from the CMFortran subroutine, it collects the results sent from the node CMMD program and finally ends. Before and after enabling the CMMD node program or invoking the CMFortran subroutine, there can be any other non-parallel computing subroutines in the sequential host program. Technically, however, this approach would pose significant difficulties on the complier (linker), as it is now the compiler’s responsibility, as opposed to that of a third-party daemon or the OS in the process-based approach described below, to manage the time-sharing of the CM5’s PNs and internal networks between the CMFortran subroutine and the CMMD node program.

B.2 Parallel Virtual Machine (PVM)

The PVM [73, 74] is a public-domain software package that supports utilization of heterogeneous networks of parallel and serial computers as a single computational resource. The PVM provides a general-purpose, message-passing interface to TCP/IP networking protocol, portable across multiple platforms.

The PVM consists of two parts: a daemon process on each host machine of the virtual machine, and a user library that contains primitive routines for initiating processes on other hosts to allow communication and data transfer between processes and for configuration of the virtual machine. The PVM assumes that each host in a virtual machine is directly connected to each other host via IP. The daemons communicate with one another using UDP sockets. TCP sockets are used between a daemon and the local PVM tasks, and also directly between tasks on the same host or on different hosts if necessary. The message-passing system is implemented on top of the UDP and TCP sockets.

The PVM software package has no built-in visualization capability. To develop a graphical user interface (GUI) for a networking application program written in PVM, one has to link to other graphics packages such as Xlib or X Toolkit. Most previous work in developing a GUI component in a PVM system was focused on performance monitoring, graphical debugging, and visual development tools for aiding the programmer in the tasks of designing, compiling, executing, debugging, and analyzing a parallel program [76, 75].

B.3 Comparison of Numerical Advection Models

In this section we use a case study to demonstrate the feasibility of integrating CM-Fortran and CMMD modules on a CM5 in the proposed environment.

In fluid dynamics, the simple one-dimensional advection equation (1) with constant positive velocity serves as a basic model for the shape-conserving movement

Figure B.2: System configuration of the prototype system

of an initial distribution of fluid volume toward positive x. Since the analytic solution is known in this simple case, numerical approximations to the advective process can be critically evaluated on fundamental properties such as accuracies, stabilities, coordinate systems, and computer time and memory requirements.

\frac{∂μ}{∂t} + u\frac{∂μ}{∂x} = 0  (B.1)

where μ is the fluid velocity and u is the mixing ratio μ ≡ C/ρ in which C is the constituent density and ρ the density of the fluid.

Our case study is to create a prototype interactive computing/visualization environment in which a couple of 1D advection models are purposely implemented in

Figure B.3: A graphical user interface of the prototype system

different programming paradigms on the CM5 and the results graphically compared through a graphical user interface.

We choose from [24] five numerical advection algorithms with different diffusion, dispersion and monotonicity properties. They are Donor-cell, Partial donor-cell, van Leer, Hain and Lax-Wendroff, named after the authors of the algorithms. Donorcell is then implemented in CMFortran and van Leer in CMMD, while the rest of the models are implemented in Fortran77. All the AVS/CM interface part codes are written in C.

To demonstrate the system’s integration capability on a heterogeneous architectures, we purposely run the AVS kernel on a IBM RS/6000, the Hain module on a DEC5000, and the Lax-Wendroff module on a SUN4, while the data-parallel Donorcell module, the message-passing van Leer module and the sequential module Partial donor-cell are running on a 32-node CM5. The system configuration is shown in Figure B.2.

Figure B.3 is a screen-dumped picture showing the system parameter control panel (left), model output windows (top) and the flow network (bottom). The flow-chart-like diagram is the process configuration built by the AVS visual programming interface Network Editor, in which Advection Interface is a system control module for model input parameters steering and the graph viewer is an AVS system module for model graphical output. Both run on the same machine as the AVS kernel. The other modules correspond to the respective advection models.

This prototype environment has the following features:

1. User interactive control of the modeling. Before or during a modeling run, the user can graphically choose advection models, set model parameters, display time steps and the total number of grid points, and time steps.

2. Graphical output of the model results. The user can decide the displaying pattern, such as line, area, bar, and other graphics parameters such as title, color, etc.

3. Flexible enough to include other advection models. It can be easily extended to add other old/new models with varying velocity fields, multidimensions and non-rectangular coordinated systems, or sequential, data-parallel, or messagepassing implementations. All the models can be transparently configured to run in a distributed computing environment.

B.4 Option Price Modeling on an IBM SP2 and a

DEC Alpha Cluster

Stock option pricing models are used to calculate a price for an option contract based on a set of market variables (e.g., exercise price, risk-free rate, time to maturity) and a set of model parameters. Model price estimates are highly sensitive to parameter values for volatility of stock price, variance of the volatility, and correlation between volatility and stock price. These model parameters are not directly observable and must be estimated from market data.

We use a set of four option-pricing models in this study. Simple models treat stock price volatility as a constant, and price only European (option exercised only at maturity of contract) options. More sophisticated models incorporate stochastic volatility processes, and price American contracts (option exercised at any time in life of contract). These models are computationally intensive and have significant communication requirements. The four pricing models are: BS — the Black-Scholes constant volatility, European model; AMC — the American binomial, constant volatility model; EUS — the European binomial, stochastic volatility model; and AMS — the American binomial, stochastic volatility model.

Figure B.4 is the system configuration of a prototype interactive simulation-on-demand system for the option-price modeling application, using the AVS/PVM framework we proposed in Section 3.3 and utilizing the network infrastructure and distributed computing facility at NPAC.

The AVS kernel runs on a SUN10 workstation which acts as both an AVS server to coordinate data-flow and top-level concurrent control among remote modules and as a network gateway that links the NPAC in-house host machines locally networked by an Ethernet to the regional end-user through the ATM-based WAN (in the actual demonstration, a NYNET link to Rome Laboratory, Griffiss Airbase, NY). The ATMbased link is built around two Fore switches that operate at 155 Mbps (OC3c), while the wide area network portion of the network operates at OC48(2400 Mbps) speed.

The two parallel pricing models (EUS model and AMS model) are implemented in PVM and run, respectively, on an 8-node IBM SP2, networked by an Ethernet at the time of evaluation, and an 8-node DEC Alpha cluster interconnected by an FDDIbased GIGAswitch. They are coupled under the proposed AVS environment with the other two sequential simple models (BS model and AMC model) running on a SUN4 and a DEC5000 workstation, respectively. The nodal processor of SP2 is an IBM RISC/6000 processor running at 62.5 MHz and is one of the most powerful processors APPENDIX B. CASE STUDIES IN CHAPTER 3 139 Figure B.4: System configuration of the interactive distributed financial modeling prototype available. The DEC Alpha farm consists of eight Alpha model 4000 workstations that are supported by a high-performance networking backbone of dedicated, switched FDDI segments. The GIGAswitch provides full FDDI bandwidth and low-latency switching to every workstation in the farm.

While displayed on the end-user’s home machine, a user interface actually runs on a remote SUN4 that combines user runtime input (model parameters, network configuration) with historical market databases stored on disk, and broadcasts this data to remote compute nodes. Top-level system synchronization occurs with each broadcast.

An IBM RS/6000 is used as a file server for non graphical output of model data. In this application, model prices calculated at remote compute nodes and corresponding market data are written to databases for later analysis.

All models’ output are graphically displayed on the end-user’s home machine (a SUN10) in AVS graph viewers. Figure B.5 gives the user interface showing the simulation control panel (left), model output windows (top) and the flow network (bottom).

Notice that with the use of PVM software, only one AVS license, i.e., the one on the Sun10, is needed for this heterogeneous distributed system.

By using portable PVM software, we expect to build similar integrated parallel computing/interactive visualization applications on other MPP architectures where PVM is supported, while AVS is not available currently on host processors such as the Intel Paragon and Cray T3D.

B.5 Distributed Computational Electromagnetics

Systems

In this section we describe our work of developing a real-world electromagnetic application on distributed computing systems. A computational electromagnetics (CEM) simulation for radar cross-section (RCS) modeling of full-scale airborne is ported to APPENDIX B. CASE STUDIES IN CHAPTER 3 141

Figure B.5: The Graphical User Interface three networked workstation cluster systems: an IBM RS/6000 cluster with Ethernet connection, a DEC Alpha farm connected by a FDDI-based Gigswitch and an ATM-connected SUN IPXs testbed, and two MPP machines: an Intel iPSC/860 and a CM-5. In the cluster implementation we use a ScaLAPACK LU solver from Oak Ridge National Laboratory/University of Tennessee in the parallel implementation for solving a dense matrix that forms the computationally intensive kernel part of this application, and adopt BLACS as the message-passing interface in overall code development to achieve high portability across the three configurations. In the MPP implementation on the CM5, we use the mixtured programming models in which the matrix solver is from the vendor-supplied CMSSL, which is written in data-parallel CMFortran. The rest of the application is written in explict message passing on the CMMD. Their performance data are reported, as well as (for comparison purposes) the timing data from all the platforms implemented. This work shows that in order to achieve optimized performance on MPPs coupled with clusters, a processes-based programming framework is needed in which components and modules written in different programming models can be seamlessly combined.

B.5.1 Introduction

Traditional electromagnetic engineering simulations are largely limited by memory requirements, as well as by sequential processing time. Most electromagnetic applications on high-performance computing systems so far experimented with or implemented have been on either massively parallel processors or traditional vectorbased supercomputers [50, 51]. The advanced development in workstation cluster technology, represented by clusterable IBM RS/6000 systems and Digital’s Alpha farm with both high-performance node processors and large memory capacity, provides new opportunities and solutions to computational electromagnetic applications that are both CPU and memory intensive. Together with the emergence of portable distributed linear algebra packages such as BLACS and ScaLAPACK and portable message-passing interface such as PVM and MPI, cluster-based computational electromagnetics systems give an attractive solution to numerical simulations in engineering electromagnetic analysis and design with modest problem sizes. They achieve better cost/performance than the MPP solutions, which is especially important to real-world applications where workstation clusters are more accessible to engineers. On the other hand, MPP is the only solution for solving big CEM problems when the memory requirement cannot be fulfilled by distributed systems.

A major motivation in this related work was to port the CEM application to multiple MPP and distributed computing systems and evaluate/demonstrate their capabilities for solving this problem. Portability, as well as performance scalability, was among the top objectives of our effort [52]. With the distributed linear algebra package Scalapack [48, 53] and BLACS [47] in our parallel implementation, we were able to achieve true code portability across Intel machines (the Intel Delta, an Intel Paragon, and an Intel iPSC/860), an IBM SP-2 with either Ethernet or switch connection, and the three distributed configurations. We focused our discussion on benchmarking and performance requirement aspects of this CEM application on the experimented cluster and MPP systems.

B.5.2 The RCS Application Problem

The CEM application used in this work is a well-established CEM package from Syracuse Research Corporation (SRC) called ParaMoM, which stands for Parametric Patch Method of Moments (MoM). The most distinctive feature of the ParaMoM is its use of basis functions that conform to parametric surfaces with curvature and thus generate accurate and stable simulation results.

ParaMoM utilizes a general surface formulation. All surface integrals are performed on the curved surface rather than on a flat facet approximation, as has been traditionally done. The basis (expansion) functions are a parametric surface generalization of the well-known Rao-Witon-Glisson functions [57]. The domains of these functions are curved three-sided surface patches that conform exactly to the underlying global surface representation. With the parametric surface-patch approach as a foundation, a comprehensive set of capabilities was developed and placed in ParaMoM 1.0[78], which has the following features:

• Accepts IGES 114 and IGES 128 parametric surfaces, flat facets, and IGES 110 wires.

• Wire antenna modeling includes user-defined loads, feed points, and connections to surfaces. Wires may radiate or scatter.

• E, H, and combined-field formulations.

• Up to three symmetry planes may be used to reduce memory and computation requirements for problems with geometric symmetry.

• Impedance-sheet formulation for treatment of resistive cards (R-cards)

• Graphics-based preprocessor to perform gridding and other model preparation functions not included in a general-purpose CAD package.

B.5.3 Parallel MoM Implementation Strategy

The ParaMoM code consists of five major consequent processing phases that have different computational requirements in terms of CPU time and memory consumption, as shown in Table B.1.

Where N is the number of unknowns, i.e., the number of basis functions, it is proportional to the surface area of the target.

The role of the setup phase is to read in the geometric target description (i.e., the parametric surface representation along with the triangulation, material, and wire connection information) and to precompute specific geometry information required by the fill algorithm (including positions and tangent vectors for the surface points used in matrix fill expansion and testing integrals). Because these geometry results Table B.1: Processing Phases of Sequential ParaMoM and Their Computational Requirements

Phase  Component  CPU time Memory
1  Setup  O(N)  O(N)
2  Matrix fill  O(N^{2}) O(N^{2})
3  RHS vector fill  O(N)  O(N)
4  Matrix factor/solve  O(N^{3}) O(N^{2})
5  Far-field computation O(N)  O(N)

will not be modified during the subsequent phases, there is no data coherency concern. At this phase we used a block-cyclic partition so that each node precomputes evenly apiece of the geometry arrays and broadcasts its results to the rest of the nodes. Each node then maintains a private copy of these arrays of reasonable size (O(N)) in its local memory.

The MoM matrix filling is well suited to distributed memory systems. To better understand its parallel implementation, Program 1 gives the sequential filling algorithm in pseudo-Fortran code. The sequential algorithm has the following features, which are well utilized and highlighted in our SPMD parallel algorithm:

1. Calculation of any Z-matrix element is completely independent of other elements in the matrix and is only dependent on its memory allocation. This property makes it possible to perform the Z-matrix calculations locally “in-place” of the processors on which the Z-matrix elements are stored, and the performance of the parallel algorithm depends solely on the locality of the Z-matrix partition, not on the networking topology of physical processors.

     c loop over source patch
            Do isource = 1, nfaces

c loop over field patch
           Do ifield = 1, nfaces
c calculate integration points for patch pair (ifield,isource)
              call patch-interaction(ifield,isource)
c loop over basis functions of the source patch
              Do isrcnode = 1, 3
c look up the index in pre-defined table
                j = lookup(isource,isrcnode)
c loop over integration points of the field patch
                Do ipnt = 1, 4
c calculate potential integrals for the patch pair -- main routine
                   call pots(isource,ifield,isrcnode)
c loop over basis functions of the field patch
                   Do ifldnode = 1, 3
c look up the index in pre-defined table
                      i = lookup(ifield,ifldnode)
c calculate intermediate result
                      tmp = exp(ifield,ifldnode,ipnt)
c accumulate the Z-matrix element where cz is the N by N Z-matrix
                      cz(i,j) = cz(i,j) + tmp
                   Enddo ifldnode
                Enddo ipnt
              Enddo isrcnode
           Enddo ifield
        Enddo isource

Program 1 The Sequential Matrix Filling Algorithm

2. Determined by the characteristics of the numbering of basis functions among APPENDIX B. CASE STUDIES IN CHAPTER 3 147

Figure B.6: Program structure of the ParaMom implementation Figure B.7: Patch-patch interaction and the Z-matrix patches, calculation for a patch pair (isource, ifield) is based on the 9 associated basis functions (isrcnode = 1 : 3, ifldnode = 1 : 3) and is eventually filled into the Z-matrix as shown in Figure B.2, where j, l are column-index related to the source path (isource) and i, k are row-index related to the field patch (ifield). From Figure B.2 it is clear that a column-major blocked partition of the Z-matrix achieves the best locality for a patch pair. It guarantees that at least 6 Z-matrix elements are mapped to a processor and that the maximum number of processors used to calculate the total 9 elements is 2. In an average case when a general SSB partition is used, the maximum number of processors for a patch pair is 4. This is to say that, compared to the ideal partition in which all the 9 elements are mapped to a single processor, the only overhead for the column-major block partition is one redundant computation of the routine “patch-interaction” which takes less than 20% of the total computation for a patch pair.

3. The locality issue examined in (2) is for a single patch pair. We must also investigate the load-balance issue for all patch pairs to see if the overall Z-matrix computation is evenly distributed to all processors. Fortunately, the numerical model statistically assures that the Z-matrix computation of all patch pairs is evenly distributed. Actually, for all Z-matrix elements except those related to boundary patches, an element’s final value is accumulated from four patch pairs and from (2) we know a patch pair contributes to nine different Z-matrix elements in most cases. Our measurements in testing cases also proved that this is statistically true.

4. It becomes clear that in (2) a larger Z-matrix size gives better data locality, and in (3) a larger machine size makes a relatively better balanced load of Z-matrix among processors. Therefore, the parallel filling algorithm will guarantee a scalable performance in terms of machine size and problem size.

Our complete parallel implementation includes:

1. In the setup phase, a parallel file input for target geometry data and a parallel algorithm for precomputation in which only global broadcast operations are used.

2. An embarrassingly parallel algorithm for the Z-matrix filling that requires some redundant calculation of patch-patch interactions but no inter-node communication.

3. A similar embarrassingly parallel algorithm for the RHS excitation vectors filling that has no inter-node communication.

4. A COMPLEX parallel LU solver. Our approach is to utilize parallel matrix equation solution software developed by others. Such software includes vendorspecific commercial software such as the CMSSL matrix equation solver on the CM5 that is written in data-parallel and is highly optimized for vectorprocessing, as well as the portable, publicly available software ScaLAPACK from Oak Ridge National Lab.

5. A parallel algorithm for far-field calculation that requires only global summation operations.

Our algorithm design is focused on Phase 2 of filling the impedance matrix (Zmatrix). The major task in our implementation is to partition the computation of Z-matrix elements among processors by distributing its memory among processors.

The Basic Linear Algebra Communication Subprograms (BLACS)[54] comprise a library that provides ease of use and portability for message-passing, matrix-based operations in parallel linear algebra programs on distributed memory machines. It supports both point-point operations between processors and collective communications on a virtual two-dimensional processor grid. ScaLAPACK, the two-dimensional distributed memory version of common LAPACK[55] routines, relies on calls to the BLAS for local computation and on calls to the PB-BLAS[49] for global computation. For portability reasons, communications take place inside the PB-BLAS through calls to the BLACS.

Table B.2: Timings of parallel I/O and LU solver for N = 2000 (in seconds)

Operation  Time
Write matrix  2.3
Read matrix  1.5
LU factorization 24.5
Solve  3.2

We were fortunate in gaining early access to the ScaLAPACK library and adopted an early release of the REAL variable arithmetic version in writing our COMPLEX routines. It is important to parallelize the full applications code so as to optimize and balance the overall performance of this application [9]. The BLACS interface also provided a suitable level of abstraction for us to implement the “Setup” and “Farfield” components. The full schematic of our parallelized ParaMoM code in cluster implementation is shown in Figure B.2.

ScaLAPACK is currently not an efficient matrix solution option on the CM5, because a BLAS library that is optimized for the CM5 vector unit does not exist. Therefore, the platform-specific CMSSL matrix equation solver is being used.

The CMSSL library uses a data-parallel programming model and cannot be directly interfaced with the message-passing, matrix-filling algorithm. Therefore, in our implementation matrix filling and factoring are two distinct program modules. The high-bandwidth parallel file system on the CM5 (DataVault) is used as a buffer between the two stages. The message-passing MoM matrix-filling program fills the matrix and writes it to the file system in the format required for the factoring stage in CMSSL. The matrix is subsequently read in by the data-parallel matrix solver stage.

There is very little performance penalty for using separate program modules for the filling and factoring, because the DataVault has very high data rates. For example, Table B.2 gives the timings for a complex matrix size = 2000*2000.

For this moderate-sized problem the I/O time is small compared to the factor time. Since the I/O time scales as N*N and the LU time scales as N**3, it is evident that for large problems the I/O time will not be a limiting factor.

The initial implementation of the matrix-filling algorithm on the CM5 is being written in F77 with calls to the CMMD message-passing library. The resulting code is adequate to investigate the efficiency of the parallel algorithm as measured by the speed-up (ratio of elapsed time for one node versus elapsed time for M nodes). However, the per-node performance is poor because the F77 code cannot utilize the CM5 vector units efficiently, and the matrix-filling part is best optimized in explicit message-passing programs. As indicated in our benchmarking timing data, to optimize the overall performance of this application, a coupled CM5 and cluster solution is required so that the matrix solver achieves the best performance on a CM5 and the filling is optimized on a clustered IBM SP2. To integrate the two components written in different programming models interactively, a unified integration framework is required so that results of the matrix filling part on the SP2 can be directly fed into the LU solver running on the CM5, bypassing the buffering on disk files as currently used in the stand-alone CM5 approach.

B.5.4 Performance Comparison and Discussion

We report in this section timings of the Z-matrix filling and LU factorization parts which take more than 95% of the total execution time. The filling part is almost 50% dominated by MFLOPS and 50% by memory access, while the LU part is more than 90% dominated by MFLOP. Elapsed time for filling and both elapse time and MFLOPS for LU are listed.

The DEC Alpha farm consists of eight Alpha model 4000 workstations running at 133 MHz and with 64 MB memory per processor, interconnected by a dedicated Gigaswitch that provides full FDDI bandwidth and low-latency switching to every processor in the farm. The Ethernet-based IBM cluster has eight IBM RS/6000 model 370 processors running at 62.5 MHz each and with 64 MB memory per processor. The ATM link between two SUN IPXs operates at a peak bandwidth of 155 Mbps(OC3c).

Figure B.8: Fill Time v.s Processors (Matrix Size = 988)

Each SUN IPX has 32 MB memory and runs at 40 MHz.

Different, modest problem sizes are used to show the performance comparison and scalability. For comparison purposes, timings of this application on an Intel iPSC/860 (40 MHz, 16 MB/node) and a CM5 (25 MHz, 16 MB/node) are also listed. The Intel implementation, with BLACS linked to its NX version, has exactly the same source code as that on the clustered systems, while the CM5 implementation used a completely different approach in which the fill part was written in a messagepassing CMMD program and the LU part was a data-parallel CMFortran program using a vendor-supplied CMSSL LU solver to utilize CM5’s high-performance vector units. The CM-5 implementation used out-of-core DataVault (or SDA) disk memory to hold and transfer the Z-matrix between the two different components. For some problem sizes, more processors have to be used on the iPSC/860 and CM-5 due to the memory requirement. Notice that a block size N_{b} = N/N_{p}, P = 1 and Q = N_{p} are used for all the tests reported here, where N_{p} is the total number of processors used, N the matrix size and P × Q the virtual processor grid. As discussed in Section 3, this partition is optimized for the matrix fill algorithm on all the platform and good for the LU part on the three distributed systems, but it is not optimized for the LU solver on the iPSC/860.

Table B.2 gives an overall breakup of timings that indicates the two dominated components and relatively balanced results. In Tables B.3 and B.4, the clustered systems show compatible or even better performances than the MPP platforms with a comparable number of processors, although the LU part is poorer, limited by the networking bandwidth and high latency of message-passing protocols on the cluster systems. The LU performance on the ATM-based configuration (Tables B.5 and B.6) is extremely poor and non scalable with the problem size. There may be two reasons:

(1) TCP/IP instead of ATM’s own protocol is used, whose high latency and that of PVM outweigh the overall bandwidth; (2) no optimized BLAS library is utilized.

ScaLAPACK makes use of any system-supplied BLAS routines that will help performance optimization at the node level. This is important, since many of the recent parallel architectures employ vector operations at the node level, and processors such as the i860 employed by the Intel iPSC/860 are well endowed with vector registers and can carry out BLAS operations very efficiently. Furthermore, even on the superscalar nodes used by machines like the IBM RS/6000 cluster, the BLAS operations provide a valuable level of abstraction for node level optimization.

The scalability of fill and LU parts is shown in Figure B.5.4.

B.5.5 A Scheduling and Visualization Framework for the RCS Simulation on a Meta-Computer

The high portability achieved in the parallel code development across multiple parallel platforms, along with the significant importance of an interactive visualization requirement, provides an excellent opportunity for us to develop a real-time scheduling and visualization interface for this application. The objectives of this prototype are three-folded, as follows:

1. An interactive visualization interface to allow real-time simulation parameter steering and graphical display of simulation results of far-field scattering, along with the three-dimensional geometry of the RCS target, APPENDIX B. CASE STUDIES IN CHAPTER 3 154

Table B.3: Timing Comparisons for N = 4889 (in seconds)

Platform  N_{p}  Fill  LU LU (mflop) Setup RHS+Field Total
Alpha(FDDI)  8 1420.0 1119.9  295.5  12.3  18.0  2570.2
IBM RS(Ethernet) 8 1500.7 1804.7  183.4  51.4  28.2  3385.0
Intel iPSC/860  64 525.6 280.9  1178.8  45.4  53.0  904.9
CM-5  32 3295.2 170.7  1938.8  21.1  4.3  3491.3

Table B.4: Timing Comparisons for N = 3060 (in seconds)

Platform  N_{p}  Fill  LU LU (mflop)
Alpha(FDDI)  8 450.4 713.5  91.6
IBM RS(Ethernet) 8 495.1 1210.3  54.0
Intel iPSC/860  16 830.9 175.6  372.1
CM-5  32 1207.0 79.5  821.9

Table B.5: Timing Comparisons for N = 1504 (in seconds)

Platform  N_{p}  Fill  LU LU (mflop)
Alpha(FDDI)  8 114.9 76.2  110.5
IBM RS(Ethernet) 8 127.1 131.8  63.9
Intel iPSC/860  8 478.6 61.5  136.9
CM-5  32 358.9 11.0  824.3
SUN(ATM)  2 2087.4 1941.4  4.3
Alpha(FDDI)  2 411.0 278.6  30.2
IBM RS(Ethernet) 2 475.6 262.0  32.1
Intel iPSC/860  2 1950.7 93.9  89.7

Table B.6: Timings of Different N on 2 SUN IPXs Interconnected by an ATM Link

N Fill LU LU (mflop)
1504 2087.4 1941.4  4.3
988 981.1 560.6  4.3
672 449.3 176.4  4.5
468 229.2 60.7  4.3
368 147.8 29.9  4.1
187  45.4  4.0  4.1

2. A high-level scheduling mechanism to facilitate runtime allocation of simulation job among multiple parallel systems available for this application.

3. A single integrated framework to couple all the components required by this prototype system that are executed under a locally-networked environment, including a GUI interface for system control and visualization, a runtime scheduling interface for parallel job allocation, and the parallel simulation job to be executed on one or more of the scheduled parallel or distributed computers.

We envision using the proposed dataflow integration framework as an integrator to facilitate all the components.

Figure B.9 gives a skeleton of a flow diagram for the framework, in which the scheduling factor is platform-dependent and is calculated based on the following common performance parameters:

1. Simulation-related parameters: (for current determined RCS target) in-core memory, number of flop counts (or CPU, memory measurements)

2. Platform-related parameters: (for current meta machine in place) clock rate, theoretical peak performance, number of processors, in-core memory size/node, APPENDIX B. CASE STUDIES IN CHAPTER 3 156 Figure B.9: A Integrated Scheduling, Visualization and Simulation Framework on a Meta-Computer communication bandwidth, (parallel) I/O bandwidth and memory bandwidth (in general, CPU, memory, I/O, and networking)

3. Current runtime system load parameters: (determined at runtime) system status (up/down), number of users, free memory available, load average (uptime, the average number of jobs in the run queue over the last 1, 5 and 15 minutes)

4. User runtime defined parameter: by default all values are 1. User can set it via a GUI to define the priority machine. Simple or sophisticated scheduling functions can be enforced. For example, here is a simple scheduling scheme:

f_{i}(a, b, c, d) = \{ \begin{matrix}10% ∗ a + 20% ∗ b + 30% ∗ c + 40% ∗ d & if the machine is up \\ 0 & otherwise \end{matrix}

where a,b,c is a parameter value calculated from the above 1,2,3, respectively. d is a user-defined number that can be graphically steered from each module’s control parameter panel associated with the AVS remote module, and 0 <= a, b, c, d <= 1. For the filter module, the decision is based on Max(f_{1}, f_{2}, f_{3}, f_{4}, f_{5}).

Figure B.10 is the GUI of this prototype showing an EM scattering field from the full-scale airborne system, which is generated by the ParaMOM MPP code on an Intel Paragon, while Figure B.11 also shows the runtime parameters control panel on the GUI.

APPENDIX B. CASE STUDIES IN CHAPTER 3 158

Figure B.10: A graphical output of the prototype system (a VFY) Figure B.11: The graphical user interface of the prototype system (a sphere

Appendix C

Case Studies of Distributed Data

Visualization and Parallel

Processing in the Software

Integration Framework

This appendix details two application case studies used in Chapter 4 to demonstrate the integration of interactive distributed data visualization and parallel simulation in a unified dataflow-based programming framework. The two case studies are chosen from financial modeling and computational electromagnetics. For each application we describe a prototype system that illustrates system configuration, software integration, performance, functionality and other issues.

C.1 A Financial Modeling Application

C.1.1 Problem Description

Stock options are contracts that give the holder of the contract the right to buy or sell the underlying stock at some time in the future. Option contracts are traded just as stocks are traded, and models that quickly and accurately price option contracts are valuable to traders and financial managers. Speculators participate in the option market to capture potentially high profits with relatively small investment capital. Financial managers buy and sell options to hedge risk in their investment portfolios.

We use a set of four option pricing models in this study. Simple models treat stock price volatility as a constant, and price only European (option exercised only at maturity of contract) options. More sophisticated models incorporate stochastic volatility processes, and price American contracts (option exercised at any time in life of contract) [38, 39]. These models are computationally intensive and have significant communication requirements. The four pricing models are: BS — the Black-Scholes constant volatility, European model; AMC — the American binomial, constant volatility model; EUS — the European binomial, stochastic volatility model; and AMS — the American binomial, stochastic volatility model. Detailed descriptions of these four models can be found in [38, 10].

C.1.2 System Configuration and Integration

Using the stock option-price modeling application as a case study, we demonstrate a simple, effective, and modular approach to coupling network-based concurrent modules into an interactive remote visualization environment as proposed in Chapter 4.3.1. Two prototype simulation on-demand systems are developed to demonstrate the portability of this framework in which parallel option pricing models are locally implemented on two system configurations (two meta machines): one with two MPP machines — a 32-node CM5 and a 8K-node DECmpp-12000; the other with two distributed systems — an Ethernet-based IBM SP2 and a cluster of DEC Alpha workstations networked by a FDDI-based GIGAswitch. They are coupled with an interactive graphical user interface over the NYNET ATM-based wide area network. AVS is used to integrate massively parallel processing, workstation-based visualization, an interactive system control, and distributed I/O modules.

Configuration 1 — NYNET + CM5 + DECmpp-12000 + Workstations

Figure C.1 is the system configuration of the first prototype interactive simulationon-demand system for the option price modeling application, utilizing the network infrastructure and supercomputing facility at NPAC.

The AVS kernel runs on a SUN10 workstation, which acts both as an AVS server to coordinate data-flow and top-level concurrent control among remote modules and as a network gateway that links the NPAC in-house host machines locally networked by an Ethernet to the regional end-user through the NYNET. The ATM-based link is built around two Fore switches that operate at 155 Mbps (OC3c) while the wide-area network portion of the network operates at a speed of OC48(2400 Mbps).

Our heterogeneous computing system for stock option pricing consists of four compute nodes, a home machine, and two file server machines. All workstations, including the front-ends of the DECmpp-12000 and CM-5, are connected by a 10MBit/second Ethernet-based LAN.

The four option pricing models run on remote compute nodes: BS model on a DEC5000, AMC model on a SUN4, EUS model on a CM-5 and AMS on a DECmpp-12000(SX). Each remote compute node has its own I/O capability. Our DECmpp-12000 is a massively parallel SIMD system with 8192 processors. Each RISC-like processor has a control processor, 40 32-bit registers, and 16 KBytes of RAM. All the processor elements are arranged in a rectangular two-dimensional grid and are tightly coupled with a DEC5000 front-end workstation. The theoretical peak performance is 650 Mflops DP. Our CM-5 is a parallel MIMD machine with 32 processing nodes. Each processing node consists of a SPARC processor for control, four proprietary vector units for numerical computation, and 32 MBytes of RAM. The control node of the CM-5 is a SUN4 workstation. The theoretical peak performance is 4 Gflops. Sequential compute nodes include a DEC5000 and a SUN4. The DEC5000 performs at 6.8 Mflops, and has 16 Mbytes memory. The SUN4 runs at 4.3 Mflops and has 32 Mbytes memory.

APPENDIX C. CASE STUDIES IN CHAPTER 4 162 Figure C.1: System Configuration 1 for the Financial Modeling on a CM-5 and a DECmpp-12000

figure=chapter4-visualization/fingui.ps,height=3.0in,width=5.5in

Figure C.2: The graphical user interface for the option price modeling (1)

While displayed on the end-user’s home machine, a user interface actually runs on a remote SUN4 that combines user runtime input (model parameters, network configuration) with historical market databases stored on disk, and broadcasts this data to remote compute nodes. Top-level system synchronization occurs with each broadcast.

An IBM RS/6000 is used as a file server for non graphical output of model data. In this application, model prices calculated at remote compute nodes, along with market data, are written to a database on the output file server for later analysis.

In summary, the heterogeneous computing system illustrated in Figure C.1 provides distributed computing, distributed memory, and distributed input/output for the stock option pricing application.

Our heterogeneous computing system integrates diverse functions, computation, visualization, and system control over a diverse set of hardware. We use a mix of programming languages on the remote compute nodes-Fortran77 on the DEC5000, C on the SUN4, CMFortran on the CM-5, and MPL (data parallel C) on the DECmpp-12000. At the operating system level, all remote modules are compiled and linked as stand-alone programs. Input and output ports are defined in modules by the programmer using specific library routines provided by AVS. Each module represents a process.

There are two sources of input data: historical market data read from disk files, and runtime input of model parameters by the user through a GUI on the home machine. Output from all four models is rendered in a graphics window of the GUI and displayed numerically in a shell window, both on the home machine, and written to a database on the file server.

Figure C.2 illustrates the GUI for managing user runtime input and output, and the system configuration. Runtime input includes user-defined model parameters and system execution modes (continuous, step or pause). Outputs include twodimensional displays of model and market prices calculated by the compute nodes. The system configuration includes choice of pricing models, network configurations, and interface layouts.

Pricing models are extremely sensitive to model parameters for implied volatility, variance of stock volatility and correlation between stock price and volatility. These parameters may be read from data files (historical estimates), calculated just prior to running the pricing models (by turning on the optimization button from the GUI), or defined at run time (by expert user’s parameter steering from the GUI).

Configuration 2 — NYNET + IBM SP2 + DEC Alpha Farm + Workstations

Figure C.3 is the system configuration of the second prototype using the same visualization/computing framework while the two parallel pricing models now ported to two clusters instead of the two MPPs in the previous configuration.

The two parallel pricing models (EUS model and AMS model) are implemented in PVM and run, respectively, on an 8-node IBM SP2, networked by an Ethernet at the time of evaluation, and an 8-node DEC Alpha cluster interconnected by an FDDIbased GIGAswitch. They are coupled under the proposed AVS environment with the other two sequential simple models (BS model and AMC model) running on a SUN4 and a DEC5000 workstation, respectively. The nodal processor of SP2 is an IBM RISC/6000 processor running at 62.5 MHz. The DEC Alpha farm consists of eight Alpha model 4000 workstations that are supported by a high-performance networking backbone of dedicated, switched FDDI segments. The GIGAswitch provides full FDDI bandwidth and low latency switching to every workstation in the farm.

All models output is graphically displayed on the end-user’s home machine (a SUN10) in AVS graph viewers. Figure C.4 is the screen dump of the GUI which is the same as that in Figure 2 with a different layout that shows the simulation control panel (left), model output windows (top), and the flow network (bottom).

APPENDIX C. CASE STUDIES IN CHAPTER 4 165 Figure C.3: System Configuration 2 for the Financial Modeling on an IBM SP2 and a DEC Alpha Farm over NYNET APPENDIX C. CASE STUDIES IN CHAPTER 4 166

Figure C.4: The graphical user interface for the option price modeling (2)

price model/machine  m_{k}  c_{k}  d_{k}  r_{k}
(M_{I} ⇒ M_{k}) (M_{k} ⇒ M_{h})
BS/DEC5000(M_{1})  0.015  0.017  0.1  0.9
AMC/SUN4(M_{2})  0.085  0.017  0.1  0.9
EUS/8K DECmpp 12000(M_{3}) 0.075  0.017  0.1  0.9
EUS/SUN4(M_{3})  4.05
AMS/32-node CM5(M_{4}) 0.025 0.017 0.1 0.9
AMS/8K DECmpp 12000(M_{4}) 0.045
AMS/SUN4(M_{4})  4.25

Table C.7: Experimental timings for the option price modeling (in seconds)

C.1.3 Performance Analysis

Timings observed for the system configuration in Figure C.1 are listed in Table C.7. According to our analysis in Chapter 4.3.2, the expected T_{d} = c_{1} + m_{1} + d_{1} + r_{1} + r_{2} + r_{3} + r_{4} = 0.017 + 0.015 + 0.1 + 4^{∗}0.9 = 3.642 seconds (assuming the scheduled order is c_{1}, c_{2}, c_{3}, and c_{4}). This value will vary because the system runs under a resource-and-time-sharing environment.

To compare the performance of the distributed configuration and a single-machine system, we also list timings from a SUN4 workstation in Table C.7. Thus, T_{s} = 0.015 + 0.085 + 4.05 + 4.25 + 4^{∗}0.9 = 12.0, and we calculate an expected speed-up of S = 12.0/3.642 = 3.3.

C.2 A Computational Electromagnetic Application

C.2.1 Problem Description

Electromagnetic scattering is a widely encountered problem in electromagnetics [32, 69, 77], with important applications in industry such as microwave equipment, radar, antenna, aviation, and electromagnetic compatibility design. Figure C.6 illustrates the EMS problem we are modeling. Above an infinite conductor plane there is an incident EM field in free space. Two slots of equal width on the conducting plane are interconnected to a microwave network behind the plane. The microwave network represents the load of waveguides — for example, a microwave receiver. The incident EM field penetrates the two slots which are filled with insulation materials such as air or oil. Connected by the microwave network, the EM fields in the two slots interact with each other, creating two equivalent magnetic current sources in the two slots. A new scattered EM field is then formed above the slots. We simulate this phenomena and calculate the strength of the scattered EM field under various physical circumstances. The presence of the two slots and the microwave load in this application requires simulation models with high-performance computation and communication. Visualization is very important in helping scientists to understand this problem under various physical conditions.

In previous work, data-parallel and message-passing algorithms for this application were developed to run efficiently on massively parallel SIMD machines such as the Connection Machine CM-2 and the DECmpp-12000, and MIMD machines such as the Connection Machine CM-5 and the iPSC/860. The data-parallel algorithms run approximately 400 times faster than sequential versions on a high-speed workstation [71]. Parallel models on high performance systems provide a unique opportunity to interactively visualize the EMS simulation in real time. This problem requires a response time of a simulation cycle that is not possible on conventional hardware.

Figure C.6 also shows the physical parameters of the electromagnetic scattering problem.

C.2.2 Parallization of the EMS Simulation Model

Components of our distributed model for simulation and visualization are:

1. Decomposition of visualization computation

2. Selection of simulation parameters APPENDIX C. CASE STUDIES IN CHAPTER 4 169

Figure C.6: Physical parameters of the EM scattering problem 3. Design of a graphical user interface

4. System integration and synchronization

5. Interstage communication and performance analysis

Using the EMS simulation, we next examine the decomposition of visualization computation and the selection of simulation parameters. The remaining components are discussed in later sections.

Computational Task Decomposition

Control parallelism and data parallelism are used to target the distributed computing model illustrated in the preceding section and an MIMD parallel machine, along with a number of high-speed workstations. We use the following guidelines for the decomposition of our model’s control parallelism:

1. Hardware architecture and computational power best suited to decomposed subtasks

2. Logical components of the application’s computational model

3. Performance balance among decomposed modules and

4. Communication requirements

The moment method [67, 68, 70] is used as the numerical model for the EMS problem, which can be represented as:

{[ Y^{a} ] + [ Y^{b} ]}V⃗ = I⃗

[ H ] = L{f (V⃗ , M⃗, [ H_{0}^{2} ])}

where [ Y^{a} ] : equivalent admittance matrix of the free space; [ Y^{b} ] : equivalent admittance matrix of the microwave network; V⃗ : coefficient vector; I⃗ : the excitation vector; M⃗ : a vector of mode functions; [ H_{0}^{2} ] : matrix of Hankel functions; f : a function; L : a linear operator on f; [ H ] : final matrix of the simulated EMS field strength.

From the previous parallel algorithm design, we observed that:

1. Calculations of [ Y^{a} ], [ Y^{b} ], I⃗, M⃗, and [ H_{0}^{2} ] can be done independently.

2. Computation of [ Y^{a} ], [ H_{0}^{2} ], and the linear solver for V⃗ have significant communication requirements and are computationally intensive.

3. [ Y^{b} ] is a sparse matrix and calculation of M⃗ requires little time. Calculation time for [ Y^{a} ], [ Y^{b} ] and I⃗ are relatively balanced.

Thus we partition computations of this application into four loosely coupled computing modules (called EM-1-SUN, EM-2-SUN, EM-3-CM5 and EM-all-CM5 in the AVS module network of Figure C.8). Three modules can run simultaneously in the distributed computing environment (see Figure C.7).

At the second level, i.e., data decomposition, because most computations of [ Y^{a} ], V⃗ (a linear solver), [ H_{0}^{2} ], and [ H ] are matrix manipulations, data parallel algorithms are developed in Fortran90 and tailored to run on the Connection Machine 5 to take advantage of the CM5’s balanced data network and control network. The CM Scientific Subroutine Library (CMSSL) is used in the data-parallel implementation [71].

At a more general level, we can view the entire system as a “metacomputer” that makes use of both functional parallelism and pipelining. In this application, functional parallelism consists of graphical I/O (i.e., user interaction, 3D rendering) and decomposed simulation computations that are handled concurrently by different components of the metacomputer. Pipelining combines calculations and communications among different processors or groups of processors (e.g. the CM5) that are carried out simultaneously in consecutive stages of the simulation. We will discuss pipelining execution of the system in later sections.

Selection of Simulation Parameters

Simulation parameters are implemented as control widgets within a graphical user interface to provide a visual medium that allows the user to interact with the simulation and visualization at runtime. We choose parameters representing the physical foundation of the EMS problem and the computational model that the user can manipulate in order to visually understand the problem.

As shown in Figure C.6, the following parameters are graphically represented to allow the user’s runtime input:

1. Φ : angle of the incident EM field(wave)

2. i : i × i is the simulated area of the EMS field

3. w : width of the slot

4. d : distance between the two slots

5. ϵ : permittivity of the media in the slot

6. μ : permeability of the media in the slot

7. h : height of the slots

8. y_{11}, y_{12}, y_{21}, y_{22} : admittance parameters of a given microwave network

In addition, four parameters represent the characteristics of the equivalent magnetic current originally formed by the penetrating incident EM field. These parameters are used as mode function expansions and the number of pulse functions in the moment method. Another parameter used for visualization purposes is the number of grid points for discretizing the visualized EMS area. All five parameters have direct impact on the computational requirements and simulation resolution of the moment method. Using different combinations of the parameters, we can visualize the EMS simulation within a large number of real application circumstances. We also use visualization of typical parameters to verify the EMS computational model and some well-known theories about this physical phenomena.

C.2.3 Implementation of the Remote Visualization Environment

System Configuration and Integration Using AVS

Figure C.7 illustrates the system configuration and module components distributed over the network to three high-end workstations and a supercomputer Connection Machine 5. The network is a ten MBit/s Ethernet-based local network. AVS is used to provide the sophisticated 3D data visualization and system control functionality required by the simulation. We use AVS to facilitate high-level networking and data transfers among visualization and computational modules on different machines in the system. AVS provides a data-channel abstraction that transparently handles type-conversion and module connectivities. This software system is optimized for data movement by using techniques such as shared-memory message passing among modules on the same machine. Message passing occurs at a high level of data abstraction in AVS. This approach helps to make optimal use of both the high-performance computing resources and the rendering capabilities of the local graphical workstation. The transparent networking capabilities of AVS open up possibilities for visualization far beyond traditional graphics capabilities [1, 30].

The local machine in our system is an IBM RS/6000 with a 24-bit color GTO APPENDIX C. CASE STUDIES IN CHAPTER 4 174

Figure C.7: The System configuration for the EMS application graphics adaptor. An AVS coroutine module (in C) on the local machine serves as a graphical input and system control interface to monitor and collect user runtime interaction with the simulation through keyboard, mouse and other I/O devices. The AVS kernel also runs on the local machine, coordinating data flows and control flows among AVS (remote) modules in the network.

We use an AVS system module called “geometry viewer” along with other system modules (“generate colormap,” “color range,” and “field to mesh”) for 3D rendering operations.

The computationally intensive modules of this application are distributed to the CM5 [28], an MIMD supercomputer that is configured with 32 processing nodes at NPAC. Each processing node (PN) of the CM5 consists of a SPARC processor for control and non-vector computation, four vector units for numerical computation, and 32 MB of RAM. It also includes a Network Interface chip that gives the node access to the CM5 internal Data Network and Control Network. The two internal networks connect all the PNs with a control processor (CP) that runs a custom version of SunOS on a SPARC host.

Two Sun SPARC workstations are used in our distributed visualization environment to run the computational modules with modest communication requirements.

All modules other than those on the local machine are implemented as AVS remote modules. Their input/output ports are defined by specific AVS libraries for receiving/sending data from/to other (remote) modules via socket connections. This configuration allows the interrupt-driven user interface input mechanisms and rendering operations to be relegated to the graphical workstation, while the computationally intensive components run on the CM5 coupled with the two workstations. This distributed simulation environment implemented in AVS provides a transparent mechanism for using distributed computing resources along with a sophisticated user interface component that permits a variety of interactive, application-specified inputs.

The flow-chart diagram in Figure C.8 (lower right) is a module network configured by the AVS Network Editor for this application.

APPENDIX C. CASE STUDIES IN CHAPTER 4 176 Figure C.8: A Graphic user interface for the EMS visualization system The Graphical User Interface

The graphical user interface includes a main control panel, three individual input panels, and a 3D rendering window.

The main control panel provides the user with parameters input and simulation control at runtime. There are seven dial widgets representing the simulation parameters used by all computing modules on the three remote machines, and a control button for starting a new simulation cycle(see lower left in Figure C.8). The rest of the simulation parameters discussed in Section C.2.2 are implemented as dial widgets in individual panels associated with only the modules requiring them to minimize redundant data transfers between the input interface module on the local machine and computational modules on remote machines. They can also be turned on from the AVS Network Control Panel (not shown in Figure C.8).

Using the AVS Geometry Viewer, 3D simulation data can be rendered in various forms such as move, rotate, scale, move-the-eye-point and perspective view, and with sophisticated rendering techniques such as lighting and shading, multiple camera, Zbuffering, 2D and 3D texture mapping, automatic removal of hidden surfaces, sphere rendering, etc. The geometry viewer also takes advantage of the hardware rendering capabilities of the GL library and the GTO graphics adaptor of the IBM RS/6000. Figure C.8 (upper) shows a typical AVS Geometry Viewer window with two cameras from two different angles.

In addition, the AVS Network Editor provides a visual programming interface enabling the user to interactively reconfigure and reuse network modules. The Layout Editor in AVS allows easy and complete customization of the control panels in the GUI. Figure C.8 illustrates this interface.

C.2.4 Interstage Communication and Synchronization

Using the data-flow programming model in AVS, message passing is identical among modules on the same machine and on different machines and is completely transparent to the module programmer. The AVS kernel (protocol) supervises data transfer, which is eventually carried out by TCP/IP at a lower level. The module programmer needs only to define module input and output ports in AVS predefined data types. Message passing among AVS modules occurs only through I/O ports. A set of routines for initializing and describing modules to AVS, as well as parameter handling, accessing data, error handling, and coroutine event handling are provided. Data sources and destinations can be flexibly defined by visually connecting module input and output ports using the Network Editor.

In most cases, we use an AVS “field” as the transferred data type, which is actually a C structure. In the module network shown in Figure C.8, data transmissions are overlapped with simulation computations (pipelining). For example, the computation of an EM-3-CM5 can be pipelined with the data transfer between Input-interface-IBM and EM-2-SUN or EM-1-SUN. Message passing within a machine is implemented in AVS by copying pointers to the same memory (shared-memory). Thus there is no network communication cost for transferring the matrix [ Y^{b} ] between the two modules, i.e., EM-3-CM5 to EM-all-CM5, on CM5’s control processor. Instead of generating a complete 3D data on the CM5 and sending them to rendering modules on the local machine, we use a computing module EM-3D-IBM on the local machine to generate the X-Y coordinate data. Only the Z-coordinate data (i.e., [ H ]) and two scalar parameters for defining the X-Y data are transferred from the CM-5 to the local machine.

Simulation parameters are defined as input ports and connected to widgets on control panels supervised by the X-window manager on the local machine. By distributing parameters only to the modules that require them, we minimize recomputation of modules in a new simulation cycle. For example, changing of a parameter on the control panel of EM-1-SUN will not activate the EM-3-CM5 and EM-2-SUN.

This system is designed to work under a complete resource and time sharing environment (shared Ethernet, CM5 and all remote workstations), thus wall-clock time for computation and communication of all modules is difficult to predict. Concurrent control in the module network plays an extremely important role in assuring correctness, robustness and reliability. We issue concurrent controls in three different places:

1. In the main control panel of the GUI a one-shot control button is set to allow the end-user to control the start of a new simulation cycle.

2. Broadcast of parameters from the module input-interface-IBM to the computing modules on the remote machines is performed only when all the other module processes in the network are inactive.

3. Module EM-all-CM5 is implemented as a coroutine such that computations of the linear solver for V⃗ and [ H ] will not be activated until all the required input data for [ Y^{a} ], [ Y^{b} ] and I⃗ have been recieved ([ Y^{a} ] if only parameters on control panel of the EM-1-SUN are changed).

C.2.5 Performance Analysis

Our experiments show that under a typical working environment (only 0.5 MBits/s of the Ethernet’s ten MBits/s capacity are available), a complete simulation cycle takes about eight seconds. This response time is quite satisfactory for this application. Table 2 lists the timing data of major system components. For comparison, timings of sequential implementation on a SUN4 workstation of the two parallel modules are also given in the table.

Calculation Comm. time w/ Comm. time w/ Comm. time w/
Module  time  ’Input-interface-IBM ’EM-all-CM5’  ’EM-3D-IBM’
EM-1-SUN 0.1(Sun)  0.02  0.045  n/a
EM-2-SUN 0.6(Sun)  0.02  0.5  n/a
EM-3-CM5 1.8(CM5)  0.02  0.0  n/a

1260(Sun)

EM-all-CM5 2.1(CM5)  n/a  n/a  0.5

120(Sun)

Rendering  3.5  n/a  n/a  n/a

Table C.8: Timings of calculations and communications for the EMS (in seconds)

Appendix D

Case Studies of Integrating World

Wide Web and Distributed

Computing

This appendix details three application case studies used in Chapter 5 to demonstrate the integration of distributed WWW and centralized high-performance parallel relational database servers in a client-server framework. The three case studies are chosen from the email-based WWW collaboration system, full-text information retrieval, and financial modeling. For each application we describe a prototype system that illustrates system configuration, software integration, performance, functionality, and other issues. Some background information about parallel Oracle 7 RDBMS software and the IBM SP2 are also given to help a reader understand the approaches we adopted.

D.1 Parallel Oracle 7 RDBMS Servers on IBM

SP2

Oracle 7 on an IBM SP2 includes these components:

1. An IBM RS/6000 SP system, including multiple processor nodes, a high-performance switch (HPS), and a control workstation

2. IBM Virtual Shared Disks (VSDs) with data striping (HSD)

3. Oracle Unix Distributed Lock Manager (DLM)

4. Oracle Parallel Server

Figure D.1 shows the relationships among Oracle instances, VSDs and the DLM on an IBM SP2 system.

D.1.1 The IBM SP2

The IBM SP2 is a distributed memory MIMD system that is scalable from 4 to 512 processor nodes. Each node is a RISC/6000 processor with at least 64MB of memory, 2GB of direct access storage devices (DASD), and an Ethernet adapter for connecting to a RISC/6000 control workstation.

The high-performance switch (HPS) provides a message-passing network connecting the processor nodes. It is used for message passing among nodes by the Virtual Shared Disk system, the Distributed Lock Manage, and the Oracle parallel query option. The HPS supports the TCP/IP network protocol.

An external LAN connects the SP2 nodes to a computer network, making them accessible from any network workstation over the LAN or WAN.

D.1.2 Virtual Shared Disks

A VSD gives each node access to disks attached to remote nodes as if there were local disks. All database files, redo log files, and control files of the Oracle parallel server must be stored on VSDs. Oracle instances on different nodes use VSDs to share access to disks transparently.

APPENDIX D. CASE STUDIES IN CHAPTER 5 183 Figure D.1: Parallel Oracle7 Server Architecture on an IBM SP2

Since a VSD allows multiple nodes simultaneous access to devices, the Oracle7 parallel server must provide its own distributed synchronization mechanisms to insure data integrity. This is done by the Distributed Lock Manager.

The data striping device HSD is a device driver layer above the VSD layer. It lets Oracle7 server stripe data across physical disks in multiple VSDs to reduce I/O bottlenecks and hot spots. With data striping, one I/O request reads or writes blocks of data on each of several separate VSDs. This automatically distributes or partitions data across physical disks and notes, which improves performance of the underlying disks drives.

D.1.3 Oracle Unix Distributed Lock Manager (DLM)

The Oracle Parallel Server on the SP2 makes use of a distributed lock manager (DLM) to coordinate access to data and other resources by processes running on different SP2 nodes. The DLM synchronizes the sharing of resources such as data blocks, and ensures the consistency of modified data, effectively providing uninterrupted, transparent, and reliable data sharing for Oracle instances.

The DLM maintains data consistency by limiting disk access with distributed locks. Each instance of the Oracle Parallel Server uses the DLM to communicate with other instances and to coordinate modifications of database locks.

The DLM lock services are implemented as shared-memory, user-level service calls. It distributes lock requests and lock data structure databases throughout the nodes. Deadlock detection is also distributed.

Each resource is associated with a master node, and a directory node keeps track of the associations between resources and master nodes. A lock request from a DLM client is sent to the lock manager process running on the master node for that resource. Every node maintains information in its shared memory about client processes, resources, and distributed locks on resources associated with that node.

Figure D.2 illustrates the architecture of the Oracle UNIX DLM. The distributed lock database of the DLM consists of the collection of all lock information in every

Figure D.2: The DLM Architecture

node’s local shared memory. The local shared memory also provides communications between the DLM background processes (the DLM daemon process and the DLM monitor process).

In Figure D.2, the DLM client is any process that requests a lock to access a shared resource. The DLM client on Node A sends a lock request directly to the remote DLM daemon process on Node B (arrow 1). The DLM daemon on Node B then processes the remote request and sends the result back to the DLM daemon on Node A (arrow 2); and the DLM daemon on Node A signals the DLM client about the result (arrow 3).

D.1.4 The Oracle7 Parallel Server (OPS)

The OPS gives users on SP2 multiple nodes access to a single database located on a pool of shared disks. All users on all clients within the SP2 system can run transactions concurrently against this database. It allows multiple instances of the Oracle database server to run concurrently and independently. Each runs on its own processing node with its own Shared Global Area (SGA) memory for a database buffers cache and its own set of Oracle7 background processes (including the system monitor, process monitor, and database write) for backup and recovery. Any number of Oracle7 instances can access the same Oracle7 database files and control files on disk, which collectively form an Oracle7 Parallel Server.

Features of the Oracle Parallel Server on SP2 include:

1. Data reliability through Parallel Cache Management using the DLM

2. Transaction isolation and concurrency control

3. Online recovery for high availability

4. Centralized administration of a single database

Oracle7 on the SP2 with the parallel query option also provides the following functions:

1. Parallel query

2. Parallel load

3. Parallel index creation

4. Parallel backup and recovery

Further reading about Parallel Oracle7 on the IBM SP2 can be found in [88].

D.2 A Web and Parallel RDBMS-Based Informa-

tion System for Internet Collaboration

D.2.1 Motivation

Information distribution/sharing and information exchange/remote-collaboration are two major Internet infomation services that have served the Internet community for many years. Popular systems for text-based information distribution and sharing services are FTP, Gopher, WAIS, ARCHIE etc. These systems and the established protocol standards were developed to satisfy particular, yet basic, needs for data access, distribution and searching among distributed hosts over wide area networks. For example, FTP is used to transfer data files from one machine to another and Gopher for publishing plain text files. The WWW system has integrated almost all the major Internet information distribution protocols into a single unified networking navigation environment. The integration approach by WWW is to retain existing data transfer protocols and server management, while replacing the particular client programs that were previously developed for accessing those non-HTTPD servers. A Web browser becomes a universal client program that can access information space on different kinds of information servers.

On the other hand, there are three major Internet services for remote collaboration and information exchange: mailing list, USENET news, and chat systems.

A mailing list is a group of email addresses that can all be reached by sending messages to a single address/server. Mail sent to the mail server is redistributed to all subscribers using the basic Internet email services. The list of addressees is maintained on a central mail server running an automated mailing list program such as Majordomo, ListServ, and ListProc.

The USENET news system was invented to reduce/improve the amount of resources consumed in the mailing list approach. Articles are now stored in a central repository on a receiving host instead of in each subscriber’s mailbox. A Network News Transfer Protocol (NNTP) [92] was developed for the distribution, inquiry, retrieval, and posting of news articles using a reliable stream (such as TCP) client-server model. For news distribution, the USENET system also uses a data propagation and replication mechanism [93] to reduce network traffic congestion and the overhead of network bandwidth over wide area networks by caching recently arrived and retrieved news articles on a hierarchical intermediate news server that mediates regional and local requests. An article is usually posted first on a local news server. That machine then forwards it to all its neighboring news servers. The sites receiving the incoming article stores it locally and in turn forwards the article to all their interested neighbors. This process continues until the entire USENET network has seen the article. To read articles in a newsgroup, a user connects to a local news server using a client news reader program.

While the mailing list and the USENET news system use slightly different distribution mechanisms, they adopt the same text format in electronic mail and use an asynchronized message-passing model for text-oriented collaboration. Mail is usually stored on a server one piece per file, and each consists of a series of heading text containing the name:value pairs of sender’s identification, time, email reply path, subject, category, and other attributes as specified in [91]. Another popular collaborative system is Internet Relay Chat (IRC) [96] for text-based teleconferencing over wide area networks. IRC protocol uses a synchronized message passing model for real-time message delivery and multiplexing. The execution model of IRC is also built on generic client-server architecture in which a single chat server forms a central point for chat clients to connect to and exchange messages in real time over wide-area networks. IRC messages are a stream of characters terminated by a CR-LF pair.

While the three systems are developed for particular needs of Internet collaboration, they share several common features that are also the common ground for our integration:

1. Text-based messages as data content. Email format is the basic data entity that is used to conceptually and physically carry nformation. The major body of a message is free-text with associated attributes, including sender identity, time, subject text and receiver identity.

2. Client-server model as execution architecture. Server management and data transfer protocol are the key components and software development effort in building these systems. Client programs can be quite different on different platforms and there are usually various user interfaces for different operating system environments. There is no unified client GUI that can facilitate all three collaborative modes.

3. Major functionality of these systems is focused on simple server management, reliable message distribution and standard protocol design. They lack sophisticated search and archiving capability as found in advanced RDBMS-based information systems. They rely on file systems to store and index data thus the logical representation of the collaboration information is almost the same as physical representation. Browsing through the message list is the meta-index approach to locating interest messages.

4. Data propagation and caching, as well as server replication, are common approaches to leverage server performance and network traffic requirements imposed on wide area networks.

We expect the three conventional collaboration systems will continue to have widespread use in various applications for many years to come. We believe the WWW provides opportunities to integrate them into a single unified framework. WWW is not only an integrated environment to access and distribute various information resources on the Internet, it can also be a generic interface and framework that integrates various existing and new Internet collaboration tools for remote group interaction and interactive information exchange. Instead of developing new collaborative systems within the new WWW environment, our approach is to integrate existing Internet systems and to develop a new Web-top software layer that provides an integrated access path to underlying non-WWW software systems. In this case study we build a powerful WWW collaborative information system that integrates and enhances basic functionality of the three conventional Internet collaboration systems. We use a parallel Web and RDBMS server technology as a centralized high-performance collaborative server to store, synchronize, and distribute email-based messages over the Internet. Built on this collaborative server is an RDBMS-based information management system that serves as a search and archive engine for USENET news, a mailing list and a chat system. This single server can facilitate either asynchronized USENET news and mailing articles or synchronized chat messages. On the client side, we develop HTML-based interface that can be embedded in any Web browser for basic collaborative functions, including read/navigate, search, compose, post, reply and receive messages. Sophisticated real-time search capabilities that are not possible in conventional systems are implemented, such as search by keyword, by date, by sender, etc. The integration approach used in this case study is on both client and server sides. On the client side, a unified GUI is developed so that any one of the three collaborative modes can be switched to at any time of collaboration by participants. On the server side, the same data format of email used by all three systems is stored in a single Oracle database and the server provides support for both SQL-based search in the collaborative information space and concurrent control in the collaborative threads. Overall, this prototype system provides support of bulletin board type for the USENET news and mailing list, and for the real-time IRC type of email-based chat.

D.2.2 System Configuration

Our system is powered on a 4-node IBM SP2 parallel machine. Parallel Oracle7 RDBMS software is used on the SP2 as a back-end search engine and database server. Appendix D.1 describes in detail this parallel database software system.

As shown in Figure D.3, key system components in the configuration include:

1. Parallel HTTPD servers. Each server is an NCSA HTTPD server. The parallel Web server consists of multiple independent HTTPD servers, each running on a separate IBM SP2 node. Each SP2 node has a unique IP address that is accessible to Internet via a Web client/server. Through a CGI to an Oracle RDBMS instance also running on each SP2 node, each Web server has access to the entire database content that is shared among all the Oracle instances. In this configuration, the HTTPD server becomes an Oracle client application that routes the request from Web clients to the Oracle server. Parallelism is obtained by distributing Web client requests and hyperlinks to multiple Web servers sharing the same database. This is done by a simple “round-robin” load-balance algorithm implemented on a master HTTPD server.

APPENDIX D. CASE STUDIES IN CHAPTER 5 191 Figure D.3: Configuration of the Integrated Internet Collaborative Server Initially, all clients are given a single URL that points to the master Web server; for example, www.merlin in Figure D.3. This URL is the entry CGI program to use/access this information system. Because each client’s identification, including client hostname and access time gathered from each HTTP connection, is logged in the database, all the URLs generated as a result of the initial request are redirected to the other Web servers in the parallel system, based on a round-robin URL assignment for concurrent users. Thus the user’s subsequent accesses are confined in a Web/Oracle server that may be different from other concurrent users. Therefore, database queries originated from concurrent Web clients will be handled by different SP2 nodes simultaneously to achieve a roughly linear performance scalability.

2. Parallel Oracle7 servers. Along with an HTTPD server, each SP2 node runs an identical Oracle RDBMS instance that operates on a shared database. This Oracle instance is a powerful data processing engine that back-ends the Web server to support concurrent simple or complex SQL-based queries and database transactions requested from the Web server. Each Oracle instance handles queries independently from the other Oracle instances. With the dynamic loadbalancing scheme in distributing URLs among all the Web/Oracle servers, the parallel Oracle server is expected to serve a large number of concurrent Internet users without serious server performance degradation.

3. A Shared Oracle database. The parallel Oracle on an IBM SP2 uses a parallel file system called Virtual Shared Disk (VSD) to make the distributed disks local to each SP2 node appear to be a single logical file system. This software automatically handles remote disk access without performance degradation (via a high-performance switch with a sustained bandwidth of 40MB/s). On top of the VSD system is another file system named Hashed Shared Disk (HSD). HSD transparently stripes database table content evenly across multiple VSDs to minimize disk contention and to alleviate disk bottleneck. Therefore, the Oracle database is identical on all SP2 nodes, and the aggregate database I/O performance far exceeds the speed of accessing data from a single disk. It provides an infrastructure to host a very large volume of data with high-performance I/O.

4. Data gathering programs. While the Web servers mostly provide a WWW-based search/navigation interface for Internet users to access this collaborative server and database, we rely on another set of programs to interface with the existing Internet collaboration facilities. They are small gateway programs whose tasks are to gather in real-time messages that occurr on conventional USENET news servers and mailing list servers and convert them into the Oracle database. For USENET news, a program periodically fetches newly arrived articles on a local news server via NNTP. For subscribers of particular mailing lists, the system has a program that periodically incorporates incoming emails into its mailbox and converts/inserts them into the Oracle database. Note that this gathering process must run in real-time while the server is serving the WWW interface. Updating a database usually involves a significant amount of concurrency control (e.g., table locks). To reduce its impact on overall system performance, we devote a particular SP2 node to carry out this operation, for example, node 4 in Figure D.3.

D.2.3 Search Interfaces

USENET Newsgroup and Mailing List

Access to the newsgroup and mailing-list database use the same WWW-based search interfaces. To satisfy different users’ needs, we have developed several different search interfaces that allow different search methods and styles (Figure D.4), while they all provide base functions of search-by-keyword, search-by-listing, search-by-sender, search-by-date, etc.

The frame-based search interface (Figure D.5) assumes the client Web browser can support frames in an HTML page. It combines a meta-index search frame that APPENDIX D. CASE STUDIES IN CHAPTER 5 194 Figure D.4: Search Interfaces for the USENET Newsgroup and Mailing list Database APPENDIX D. CASE STUDIES IN CHAPTER 5 195

Figure D.5: A Frame-Based Search Interface APPENDIX D. CASE STUDIES IN CHAPTER 5 196

Figure D.6: A Form-Based Search Interface APPENDIX D. CASE STUDIES IN CHAPTER 5 197

Figure D.7: A Hypermail-Like Search Interface APPENDIX D. CASE STUDIES IN CHAPTER 5 198

Figure D.8: A Calendar-based Search Interface consists of multiple search options with a message-body display on the same page to allow tight navigation of search results.

The form-based search interface (Figure D.6) ultilizes HTML form constructs to allow very specific queries for each search option.

The line-based search interface (Figure D.7) is modeled after a popular WWW mail-archive system HyperMail [127]. Almost any Web browser can use this interface without support of enhanced HTML constructs such as frame, table, and form.

The calendar-based interface (Figure D.8) combines features in the other three interfaces. In additon to the index and search widgets used in the other interfaces, it uses a calendar as the meta-index to group monthly and weekly messages for each newsgroup or mailing list.

Detailed functionality of the prototype can be found at ASKNPAC [98].

D.3 WWW Search Systems Using SQL*TextRetrieval

and Parallel Server for Unstructured Data

D.3.1 Components of a WWW search system

A Web search system is a facility to locate/discover relevant information contained in the WWW URLs. The search domain can either be the whole WWW information space (including httpd, ftp, gopher, wais and USENET news servers) on the Internet or HTML documents on a single web site.

As shown in Figure D.9, a Web search system generally has four major components: a gathering subsystem, an indexing subsystem, a search engine, and a Webbased search interface.

Web Gathering Subsystem

A Web Gathering subsystem’s major function is to gather web pages/files from either remote or local WWW servers. It may involve either the whole Internet “Web Space”

APPENDIX D. CASE STUDIES IN CHAPTER 5 200 Figure D.9: Components of a Web Search System which includes information on HTTPD, FTP, GOPHER, WAIS and USENET news servers or HTML documents on a single Web server. This gathering task is usually carried out automatically by a relatively small program sometimes called “Web robot” or “Web Spider” or “Web Agent” whose functionality can be briefly described as follows:

1. Starting from a single URL, it requests a web page from the remote web site. Upon receiving the full page, it parses and adds all the URLs in this page to a queue for further gathering.

2. For the web page from (1), according to the indexing rules predefined in the indexing subsystem and the search engine, it parses the page and filters only the information needed by the indexing subsystem.

3. After (2) is done, it gets a new URL from the queue, checks if the new URL is already indexed and proceeds with (1) again to repeat the gathering process.

In summary, a web robot continuously requests files from remote web sites and parses and filters each file for indexing and further gathering.

It is up to the gathering subsystem to decide where, what, and when the information will be cached/indexed/updated into the text database for web search. It determines the information content and volume and, to a degree, the performance of a web search system. It can also have significant impact on the accuracy of a web search system. Different web systems may use different rules and approaches. Major issues and design parameters for the gathering subsystem are:

1. From where to gather:

(a) all or only HTTPD,FTP,GOPHER,WAIS,NEWS server

(b) for a single server, all or first N levels of URLs in the ’hyperlinks tree’ (c) web pages in single or multiple text formats of HTML, Plain Text, Latex,

etc.

2. What to index:

(a) all words or parts of ’significant’ keywords in a web page, i.e., full-text or keywords only

(b) attributes of a file, such as last update date, title, subtitles, size, outline etc.

(c) controlled by size/lines, e.g., first 10 lines of a text file.

3. When to gather/index/update:

(a) the same as or a separate indexing database from the current one used by web search interface choice

(b) real time (i.e. online indexing) or batch during nights.

4. Performance of a gather: number of files that can be gathered/indexed per day/hour

Indexing Subsystem

The indexing subsystem defines how contents gathered from remote web sites is internally stored and managed in a text database in order to efficiently and effectively support searching. It depends on the logical data structure of text entities and the physical organization/layout of textual indexes in the database. The indexing subsystem is tightly coupled with the search engine as its sole purpose is to speed up the free text search/query process conducted by the search engine. A common approach is to build an “inverted index” for all keywords in a document.

The major decisions to be made for the indexing subsystems are:

1. What compression scheme will be used to store the text and its indexes.

2. Do we keep both original text and indexes in the database, or just indexes? In order to support certain advanced search capabilities such as phrase search and proximity search, the original text must be stored together with its indexes. This will increase the space consumption of the text database.

3. Index modes: real-time indexing, batch indexing, or incremental indexing. Realtime indexing allows updates, inserts or deletes of text in/from the existing text database at any time, which will automatically modify the associated indexes, without affecting the simultaneous text retrieval on the index database. Batch indexing only updates index in a batch mode after a bulk of texts has been modified/loaded. Incremental indexing allows the indexing process to be done incrementally, as only documents that have changed since the last indexing run are indexed. Thus, adding new text will not affect the existing indexes in the database.

4. Case sensitivity, symbols, or numbers. Support of case sensitivity and symbols in the index will increase the space requirements.

Performance of a web search system will largely depend on how good its index scheme is. A good index scheme should have moderate storage requirements, while significantly speeding up the query processing.

Search Engine Subsystem

The search engine subsystem accepts queries from a Web-based search interface and schedules, partitions, and executes them against the indexed database to locate URLs and their associated attributes that satisfy the query criteria. It also deals with the I/O, caching, assembling, sorting, and performance-related system activity. Based on what content is indexed and on the scheme used in the indexing subsystem, it implements/supports different search algorithms to provide various search capabilities of a text retrieval system. Together with the previous two subsystems, the search engine determines what basic and advanced search capability a system can support. It is critical for overall system performance.

Most Web search systems currently online are keyword-based search systems. The major search functions include:

Basic search capability:

1. Keywords with logical operators and their combinations, including “and,” “or,” “not”

2. Regular expression of keywords, including single/multiple wildcard matching

3. keyword expansion: stemming, fuzzy, soundex, expansions based on a thesaurus (such as synonyms)

4. Ranking of query results

5. Case sensitive/insensitive

Advanced search capability:

1. Summarizing — summarize a document

2. Similarity search - search “similar” documents to a particular document

3. Phrase search

4. Proximity search — specify words distance between keywords

A Web-Based Search Interface

Any information retrieval system has the two core subsystems, “indexing” and “search engine,” discussed above. They represent base technology developed in information processing literature and are used directly in the Web search systems. What is unique and different in a web search system, compared to a traditional text retrieval system, lies in the “Web gathering” and “Web search interface” subsystems. These two subsystems are also the major focus of the development work we have undertaken, as we rely on Oracle’s RDBMS and SQL*TextRetrieval technology to build the “search engine” and “indexing” subsystems.

A conventional text retrieval system usually has its own search interface and also allows developers to build new search interfaces to access the text database. For example, SQL*TextRetrieval provides both form-based and C-based APIs. Most text retrieval systems use a client-server model as its fundamental model to build access interface to end-users. Therefore, the client-server model used in the WWW makes it seamless and natural to integrate a text retrieval system into a web search system, where a web client becomes a client of a text retrieval system. Due to the stateless (or sessionless) feature inherent to the web client-server model, new techniques are required in a web search interface that are quite different from those used in a traditional search interface design.

The major issues in developing a web search interface are:

1. Integration of a web server and the back-end search engine, i.e., interaction between a web server and the search engine, to efficiently facilitate the single transaction path: query requests — query processing — query results. While interaction between a Web client and a Web server is handled by the common http protocol, the mechanism of the interaction between a Web server and a search engine may vary in different Web search systems. Because a Web search system is designed to handle a large number of requests (which are, essentially, OLTP read-only transactions) simultaneously, and a heavy load is assumed on the server side, high-performance server technology is usually required to support scalable CPU and I/O processing. Another important consideration is how the system handles simultaneous search and database updates/indexing in real time. Most current Web search systems use some very limited “parallel processing” techniques and replication technology to handle performance scalability issues. For instance, a common approach popularly used in Lycos, InfoSeek, OpenText etc. is to use more than one workstation, each running a Web server and a local database. The local database on each machine is identical across all the machines, replicated from the machine dedicated to gather/index new Web pages. The replication (thus the updates) of the local database usually is done once a day during the night (some once a week/month). As far as we can see, this is a far cry from modern database replication techniques.

2. Sessionless Web connections towards session-oriented database transactions on the Web search interface. In many situations, session-oriented memorizing capability is required for Web search activity. For example, due to networking bandwidth and cache/memory constraints of the client machine, one query may generate a copious output that is too costly to send back all the way down to the display of the client’s browser. One possible approach is to store all the results on the server and provide navigation buttons (next, previous, etc.) to guide the user in going through the query results in subsets. Because each user may issue a different query and the navigation requires more than one Web transaction, the Web search system must “remember” the first query’s results for each concurrent client. This means that a user’s Web session includes the first query and successive navigation transactions. In the early days of Web search systems, the only way to solve this problem was to use a “Max hits” button for the first query so that the system restricted the maximum number of query results to be returned to the client. Currently, session-oriented transaction are becoming a common practice in most Web search systems.

3. Query refining. This capability implies that after a query is issued whose search domain may be the whole indexed database, the new query uses the results returned in the previous query as its search domain. In that way, one can keep refining the search and finally find/narrow down to the required search targets.

4. Support of the “natural language like query”. Most Web search systems support only keywords and their regular expressions to form a query. A good Web search system should also allow users to type in queries using natural language. This function requires additional processing and analyzing of the query input from the search interface.

D.3.2 Oracle SQL*TextRetrieval

Traditional RDBMS systems are developed to deal with well-structured information entities such as business data. Information represented in WWW space is usually in the form of free-format with little or no predefined structure such as HTML pages, program source code, and plain-ASCII articles. Because HTML is a very loosely defined markup language, an HTML page can be loosely or even poorly formatted. For example, not every page has a title (as enclosed by ¡title¿ and ¡/title¿ tag in HTML), although it is obligatory and acceptable by any Web browser. Although documents in free-text form can be stored in tables of an RDBMS server, the search for the content of documents without predefined search terms (words, phrases, or concepts) requires a serial scan of the table and an exact pattern matching the text column. Performance of such a search is not acceptable. The standard indexing techniques used in Oracle RDBMS cannot help in this case, as the full-text documents are stored in single or multiple table columns of character type. A new indexing facility, together with its text search utility, must be developed to solve the performance issue in free text searching. SQL*TextRetrieval is one such solutions developed by Oracle Corp. SQL*TR indexes both text contained in the multimedia documents and in the textual description of the data. The OPS allows users to perform a free text search that returns a set of documents or database rows that match the selection criteria. Because SQL*TR is a software layer built on top of the core Oracle7 RDBMS engine, it uses SQL queries as the standard method and query language for retrieving both textual information and structured data. It can take full advantage of Oracle7 server features such as concurrency control, recovery, data integrity, security, backup, and server management. Users can combine searching of structured information, (such as title, author, size, date of a Web page) with free text searching. Among many utilities provided in the SQL*TR package [86] to build a fully functional yet traditional information retrieval system, the indexing facility and text retrieval API (in C) are most relevant to building a WWW search system. Readers are refered to [86, 87] for details of SQL*TR.

D.3.3 A Prototype WWW Search System

Our approach to building a Web search system was to develop a high-performance and intelligent Web-gathering subsystem and to build a flexible and customizable HTML form-based Web search interface while employing parallel Oracle RDBMS technology for the search engine and server management, and SQL*TextRetrieval for the fulltext indexing engine. Because the SQL*TR package was originally not developed for WWW information retrieval, much work has also been devoted to interfacing the gathering subsystem with SQL*TR, and to integrating the Web search interface with the parallel Oracle7 RDBMS server on an IBM SP2 parallel machine.

Figure D.10 is the system diagram of the Web-gathering subsystem developed for our Web search system. It consists of three major components: a loader, a checker and one or more gatherers, each of which is an individual process. The fundamental philosophy behind this design is that the gathering program must be both high performance (the potential performance bottleneck of the limited networking bandwidth over the wide area network should not become a bottleneck for the gathering process; the full advantage of an RDBMS server in concurrency control and caching should be utilized to build such an RDBMS-based Web agent) and intelligent. (It should detect the URLs that have different symbolic names but point to the same document, the Web robot convention [80] should be followed, and it should be extensible to gather domain/topic specific information.)

During the gathering process the loader and the checker are connected to the RDBMS server where tables consisting of the text and index database are stored. A set of temporary tables are also created to store URLs and hosts used to identify URLs that are new (to be gathered), pending, being processed, or waiting to be updated in the gathering processing by the gatherers and the checker. Concurrency control of accessing or updating these tables by the checker and the loader is managed by the RDBMS server. To sustain the maximum networking bandwidth, more than one gatherer can be started to interact with Web servers simultaneously via HTTP to get the bulk of documents in HTML. Each light-weighted gatherer parses the document APPENDIX D. CASE STUDIES IN CHAPTER 5 209

Figure D.10: An RDBMS-based Web robot system to identify URLs to be considered for further gathering and a couple of attributes associated with the document, including URL, size, date, title, number of URLs and inline images, outline (first 256 characters), etc.

Major functions of the loader include:

1. Fetch new URLs to be gathered/updated from the Oracle server and pass them to gatherers upon request.

2. Invoke SQL*TR “rtloa” process to load/update and/or index incrementally documents when they become available (in batch).

The checker is responsible for checking URLs:

1. Validate URLs to be gathered according to several filtering conditions:

(a) the disallowed URLs specified in the “robot.tx” on a host

(b) URLs with all the attributes (size, title, outline etc.) are identical yet may have different hostnames (alias) or paths (symbolic links)

(c) URLs found to be non-existed/removed in the previous visits of the gatherer (dead links)

(d) URLs being processed by the loader or one of the gatherers

2. Filter out URLs not to be gathered according to some predefined conditions, such as URLs for server statistics.

3. Filter in only URLs to be gathered according to predefined conditions, such as the online HTML books (http://www.infomall.org/npac/pcw/*), NPAC Web server (http://www.npac.syr.edu/*), or any WWW server (http://*,ftp://* etc).

Currently the information used by the checker to do the filtering is only the URL itself and the URL/host tables in the database. To make the gathering subsystem more intelligent, to be able to gather/index domain-specific information and such, the architecture design of our system is developed to be extensible/open. For example, another module “filter” between the gatherer and the loader can be added that filters documents (not URLs) based on the content of the documents.

Figure D.11 is the main search interface of the prototyping system where one of the indexed WWW databases is the NPAC official Web server. Other databases currently indexed include three online HTML books in parallel computing

Built directly on top of the text retrieval in SQL*TR, this HTML form-based interface allows users to have the following search options:

1. Logical operations of keywords — AND/OR (“all”/“any”), NOT (“contain”/“not contain”)

2. Word expansions:

(a) word stemming — expanding a keyword based on its stem and indexed words having the same stem (wildcard matching on either end)

(b) fuzzy or soundex — words similarly spelled or phonetically similar

(c) synonyms — same or nearly same meaning as other words predefined in the same synonym ring of a thesaurus. We converted a public domain Roget Thesaurus into our database that has about 6000 synonym rings

(d) search keywords to be treated as case sensitive/insensitive

3. Phrase search — exact matching of a phrase

4. Proximity search — two words with relative location to each other

The interface also provides access to a gateway to view all indexed words and stoplist words currently in the database. This provides a simple server management layer.

As shown in Figure D.12, search results are returned to the client’s Web browser one subset per page. Included in each result page are:

APPENDIX D. CASE STUDIES IN CHAPTER 5 212 Figure D.11: The search interface of the NPAC Web search system APPENDIX D. CASE STUDIES IN CHAPTER 5 213 Figure D.12: Display of search results and an interface of the navigation and refining 1. A set of navigation buttons (“next”, “previous”, “top” and “end”) to allow

the user to browse through all the query results page by page. SQL*TR’s Stored Query Expressions (SQE), together with a sequence, is used to first store the query and its hitlist (i.e., identification of matching documents) and later restore them when these buttons are activated to request additional query results in successive Web client-server connections. The use of SQE prevents the initial query from rerunning. Note that the hitlist stored by SQE is in a highly compressed bitmap form so that the temporary space consumed should not be an issue.

2. A message area indicating search keywords and their expansions, and total number of matching documents found.

3. The line items for matching documents that contain the search keywords, each of which is represented by a set of attributes associated with the document. The title of each document is hyperlinked back to the original URL for quick browsing of the full document source, while the URL attribute links to another query to pull out the document text from the database and make the search keywords to be highlighted in the display.

4. A similar search interface to allow the user to refine the previous query so that the search domain of a new query will be the output of the previous query and not the whole database. This is again implemented by SQL*TR’s SQE.

By using word expansions we attempt to broaden the search domain to improve search accuracy limited by keywords search; while using the “query refining” we support gradual narrowing of the search domain, which may be over-extended by the word expansions.

Bibliography

[1] Advanced Visual Systems, Inc. AVS 4.0 Developer’s Guide and User’s Guide, May 1992.

[2] K. M. Chandy, C. Kesselman. Compositional C++: Compositional Parallel Programming, Technical Report, California Institute of Technology, 1992.

[3] G. Cheng, K. A. Hawick, G. Mortensen, and G. Fox. Distributed Computational Eletromagnetics Systems, Proc. of the 7th SIAM conference on Parallel Processing for Scientific Computing, Feb. 15-17, 1995.

[4] G. Cheng, K. Mills and G. Fox. An Interactive Visualization Environment for Financial Modeling on Heterogeneous Computing Systems, Proc. of the 6th SIAM Conference on Parallel Processing for Scientific Computing, R. F. Sincovec, eds., SIAM, Norfolk, VA, March 1993.

[5] G. Cheng, Y. Lu, G. C. Fox, K. Mills and T. Haupt. An Interactive Remote Visualization Environment for an Electromagnetic Scattering Simulation on a High Performance Computing System, Proc. of Supercomputing ‘93, Portland, OR, November,1993.

[6] G. Cheng, and Y. Zhang. A Functional + Logic Programming Language in Interpretation-Compilation Implementation, Lisp And Symbolic Computation: An International Journal, Vol. 5, No. 3, 1992, pp. 133-156.

[7] G. Cheng, C. Faigle, G. C. Fox, W. Fumanski, B. Li, and K. Mills. Exploring AVS for HPDC Software Integration: Case Studies Towards Parallel Support for GIS, Proc. of the 2nd AVS Conference AVS’93, Lake Buena Vista, FL, May 1993.

[8] G. Cheng, G. Fox, K. Mills and Marek Podgorny. Developing Interactive PVM-based Parallel Programs on Distributed Computing Systems within AVS Framework, Proc. of the 3rd Annual International AVS Conference, JOIN THE REVOLUTION: AVS’94, Boston, MA, May 2-4, 1994.

[9] G. Cheng, G. C. Fox, and K. A. Hawick. A Scaleable Paradigm for Effectively-Dense Matrix Formulated Applications, Proc. of the European Conference and Exhibition on High-Performance Computing and Networking (HPCN Europe) 1994, Munich, Germany, 1994.

[10] K. Mills, M. Vinson, and G. Cheng. A Large Scale Comparison of Option Pricing Models with Historical Market Data, Proc. of the 4th Symposium on the Frontiers of Massively Parallel Computation, Oct. 19-21, 1992, McLean, Virginia.

[11] K. Mills, G. Cheng, M. Vinson, S. Ranka, and G. Fox. Software Issues and Performance of a Parallel Model for Stock Option Pricing, Proc. of the Fifth Australian Supercomputing Conference, Dec. 6-7, 1992, Melbourne, Australia.

[12] K. Mills and G. C. Fox. HPCC Application Development and Technology Transfer to Industry, Postproceedings of the New Frontier: A Workshop on Future Directions of Massively Parallel Processing, IEEE Computer Society Press, Los Alamitos, CA, July 1993.

[13] MPI: A Message-Passing Interface Standard, May 5, 1994, http://www.mcs.anl.gov/mpi/mpi-report/mpi-report.html.

[14] I. Foster and K. M. Chandy, Fortran M: A Language for Modular Parallel Programming, Preprint MCS-P237-0992, Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, Ill., 1992.

[15] I. Foster, Fortran M as a language for building earth system models, Preprint MCS-P345-0193, Argonne National Laboratory, and Proc. 5th ECMWF Workshop on Parallel Processing in Meteorology, ECMWF, Reading, U.K., 1992.

[16] G. Fox, Parallel Computing in Industry: An Initial Survey, Proc. of Fifth Australian Supercomputing Conference, World Congress Centre, Melbourne, Australia, December, 1992.

[17] G. C. Fox, Parallel Computers and Complex Systems, Complex Systems ’92: From Biology to Computation, Inaugural Australian National Conference on Complex Systems, December 1992. Editors: Bossomaier, David G. Green. CRPC-TR92266

[18] G. C. Fox, S. Hiranadani, K. Kennedy, C. Koelbel, U. Kremer, C-W Tseng, and M-Y Wu, Fortran D Language Specification, Syracuse Center for Computational Science-42c, Rice COMP TR90-141, 37 pps, 1991.

[19] G. C. Fox, et al, InfoVision: Information, Video, Imagery and Simulation on Demand, Technical Report, Syracuse Center for Computational Science SCCS-575, December, 1993.

[20] G. C. Fox, et al. InfoMall: A Scalable Organization for the Decelopment of HPCC Software and Systems, Technical Report, Syracuse Center for Computational Science SCCS-531, October, 1993.

[21] B. Hailpern. Multi-Paradigm Languages, IEEE Software, Vol. 3, No. 1, 1986, pp. 54-66.

[22] High Performance Fortran Language Specification, High Performance Fortran Forum, May, 1993, Version 1.0, 184 pp., Rice University, Houston, Texas.

[23] G. Oberbrunner. Parallel Networking and Visualization on the Connection Machine CM-5, the Symposium on High Performance Distributed Computing HPDC-1,September, 1992, pp. 78-84, Syracuse, NY.

[24] R. B. Rood. Numerical Advection Algorithms and Their Role in Atmospheric Transport and Chemistry Models, Review of Geophysics, Vol. 25, No. 1, pp. 71-100, February, 1987.

[25] L. L. Smarr and C. E. Catlett. Metacomputing, Communication of the ACM, Vol. 35, No. 6, June, 1992, pp. 45-52.

[26] P. A. ‘Suhler, J. ‘Biswas, K. M. Korner and J. Browne. TDFL: A Task-Level Dataflow Language, Journal of Parallel and Distributed Computing, No. 9, 1990, pp. 103-115.

[27] System Software and Tools for High Performance Computing Environments, Final report of the Workshop on System Software and Tools for High Performance Computing Environments, Pasadena, California, April 14-16, 1992.

[28] Thinking Machines Corporation, The Connection Machine CM-5 Technical Summary, Technical Report, Cambridge, MA, October 1991.

[29] Thinking Machines Corporation, CM Fortran Utility Library Reference Manual, Technical Report, Cambridge, MA, January 1993.

[30] C. Upson, T. Faulhaber, Jr., D. Kamins, D. Laidlaw, D. Schlegel, J. Vroom, R. Gurwitz and A. van Dam, The Application Visualization System: A Computational Environment for Scientific Visualization, IEEE Computer Graphics and Applications, July, 1989.

[31] Silicon Graphics Inc. Iris Explorer User’s Guide, 1992.

[32] R. F. Harrington. Time-Harmonic Electromagnetic Fields, McGraw-Hill Book Company, New York (1961).

[33] D. M. Elwood, G. I. Fann, and R. J. Littlefield. PeIGS, Parallel Eigensystem Solver, Pacific Northwest Laboratory, July 1995. available for anonymous FTP via ftp://ftp.pnl.gov/pub/peigs.

[34] Frank J. Seiler Research Laboratory, United States Air Force Academy, CO 80840. MOPAC Manual, DEC-3100 edition, December 1990.

[35] J.J.P. Stewart, P. Csaszar, and P. Pulay. Fast Semiempirical Calculations, Journal of Comp. Chem., page pp. 227, 1982.

[36] Atilla Szabo and Neil S. Ostlund. Modern Quantum Chemistry, McGraw-Hill, New York, first edition, revised edition, 1989. ISBN 0-07-062739-8.

[37] Tomasz Haupt, Tseng-Hui Lin and Geoffrey C. Fox. Parallelizing MOPAC on distributed computing systems within AVS framework, Technical Report Syracuse Center for Computational Science, SCCS-744, Northeast Parallel Architectures Center at Syracuse Unive rsity, 1995.

[38] J. Cox, S. Ross, and M. Rubinstein. Option Pricing: A Simplified Approach, Journal of Financial Economics, Vol. 7, pp. 229-63, 1979.

[39] T. Finucane. Binomial Approximations of American Call Option Prices with Stochastic Volatilities, published in Journal of Finance, 1992.

[40] http://www.npac.syr.edu/projects/madic/madic.html.

[41] http://www.npac.syr.edu/projects/nasa/home.html.

[42] J. B. Dennis and R. P. Misunnas. A Preliminary Architecture for a Basic Dataflow Processor, Proc 2nd Annual Symposium on Computer Architecture, Huston, Jan 1975.

[43] L. Bic and J. Gaudiot (Ed.). Special Issue: Dataflow Processing, J. of Parallel and Distributed Computing, Vol. 10, No. 4, Dec 1990.

[44] T. Razik and J. Jacobs Jr.. Systems Definitions and Derivation Based on Structural Data Flow Modeling, System Research, Vol. 12, No. 3, pp. 209-216, [45] S. Weerawarana, E. Houstis, J. Rice et al.. PDELab: An Object-Orientied

Framework for Building Problem Solving Environments for PDE Based Applications, Technical Report, Dept. of Computer Science, Purdue University, May 1996.

[46] E. Gallopoulos, E. Houstis and J. Rice. Computer as Thinker/Doer: Problem-Solving Environments for COmputational Science, IEEE Computational Science and Engineering, Summer 1994, pp. 11-23.

[47] E. Anderson, A. Benzoni, J. J. Dongarra, S. Moulton, S. Ostrouchov, B. Tourancheau, and R. A. van de Geijn, Basic Linear Algebra Communication Subprograms, Proc. of the 6th Distributed Memory Computing Conference, IEEE Computer Society Press, 1991.

[48] J. Choi, J. J. Dongarra, R. Pozo, and D. W. Walker. Scalapack: A scalable linear algebra library for distributed memory concurrent computers, Proc. of the Fourth Symposium on the Frontiers of Massively Parallel Computation, PP 120-127. IEEE Computer Society Press, 1992.

[49] J. Choi, J. J. Dongarra, and D. W. Walker, PB-BLAS: A Set of Parallel Block Basic Linear Algebra Subprograms, Proc. of the 1994 Scalable High Performance Computing Conference, May, 1994, Knoxville, TN, IEEE Computer Society Press.

[50] T. Cwik, J. Patterson, D. Scott, ”Electromagnetic Scattering Calculations on the Intel Touchstone Delta,” Proc. of Supercomputing ‘92, IEEE Press, 1992, pp. 538-542.

[51] T. Cwik and J. Patterson (Eds.) Progress in Electromagnetic Research: Computational Electromagnetics and Supercomputer Architecture, EMW Publishing, Cambridge, MA, 1993.

[52] Development and Implementation of Computational Electromagnetics Techniques on Massively Parallel Architectures, Vol. 1 - 5, final project reports, SRC/NPAC, Syracuse, NY, 1994.

[53] J. J. Dongarra, R. A. van de Geijn and D. W. Walker. A look at scalable dense linear algebra libraries, Proc. of the Scalable High-Performance Computing Conference, PP372-379, IEEE Publishers, 1992.

[54] J. J. Dongarra, R. A. van de Geijn and R. Clint Whaley, A Users Guide to the BLACS, Technical Report, ORNL/UTK, Nov. 1993.

[55] LAPACK Users’ Guide, E.Anderson, Z.Bai, C.Bischof, J.Demmel, J.Dongarra, J. Du Croz, A.Greenbaum, S.Hammarling, A.McKenney, S.Ostrouchov and D.Sorensen, SIAM, 1992.

[56] Parametric Method of Moments (ParaMoM) RCS Prediction Packages, Version 1.0, User’s Manual, Syracuse Research Corporation, TD92-1321, October, 1992.

[57] S.M. Rao. Electromagnetic Scattering and Radiation of Arbitrarily-Shaped Surfaces by Triangular Patch Modeling, PhD Dissertation, University of Mississippi, August 1980.

[58] Proc. of the Pasadena Workshop on System Software and Tools for High Performance Computing Environments, Pasadena, CA, April 14-16, 1992.

[59] Proc. of the Workshop and Conference on Grand Challenges Applications and Software Technology, Pittsburg, Pennsylvania, May 4-7, 1993.

[60] G. Oberbrunner. MP/Express: Parallel/Distributed Project White Paper, AVS Inc., 1995. http://www.avs.com/techpapers/xp/mp-express/white-paper.html.

[61] M. F. Krogh and C. D. Hansen, Visualization on Massively Parallel Computers using CM/AVS, Proc. of the 2nd AVS Conference AVS’93, Lake Buena Vista, FL, May 1993.

[62] A. J. Grant. Multiprocessor AVS, Proc. of the AVS UK Users group conference, Imperial College, London, 1993.

[63] A. Vaziri. Experiences with CM-AVS to Visualize and Compute Simulation Data on the CM-5, Proc. of the 3rd Annual International AVS Conference, AVS’94, Boston, MA, May, 1994.

[64] W. Kraske, C. Asano. Real Time MPP 3-D Volumetric Visualization: Medical Imaging on the Cray T3D with AVS, the 4th Annual International AVS Conference, AVS’95, Boston, MA, April, 1995.

[65] K. Woys, M. Roth. AVS Optimization for CRAY Y-MP Vector Processing, the 4th Annual International AVS Conference, AVS’95, Boston, MA, April, 1995.

[66] R. Flanery Jr., B. D. Semeraro, AVS/EXPRESS and PVM: Gas and Oil National Information Infrastructure (GO-NII) Project, the 4th Annual International AVS Conference, AVS’95, Boston, MA, April, 1995.

[67] R. F. Harrington. Field Computation by Moment Methods, the Macmillan Co., New York (1968). Reprinted by Krieger Publishing Co., Malabar, FL (1982).

[68] R. F. Harrington. Matrix Methods For Field Problems, Proc. IEEE, vol. 55, No. 2, pp. 136-149, Feb. 1967.

[69] E. C. Jordon and K. G. Balmain. Electromagnetic Waves and Radiating Systems, Second Edition, Prentice-Hall, Inc., Englewood Cliffs, New Jersey (1969).

[70] Y. Lu and R. F. Harrington. Electromagnetic Scattering from a Plane Conducting Two Slots Terminated by Microwave Network(TE Case), Technical Report, TR-91-2, ECE Department, Syracuse University, August 1991.

[71] Y. Lu, A. G. Mohamed, R. F. Harrington, Implementation of Electromagnetic Scattering From Conductors containing Loaded Slots on the Connection Machine CM-2, Technical Report, Syracuse Center for Computational Science 270, March, 1992, also CRPC-TR92209.

[72] G. A. Mohmad, G. Fox, G. Laszewski, M. Parashar, T. Haupt, K. Mills, Y. Lu, N. Lin, and N. Yeh, Applications Benchmarking Set for Fortran-D and High Performance Fortran, Technical Report, Syracuse Center for Computational Science 327, June, 1992, also CRPC-TR92260.

[73] V. Sunderam. PVM: A Framework for Parallel Distributed Computing, Concurrency: practice and experience, 2(4), Dec. 1990.

[74] G. A. Geist, A. Beguelin, J. Dongarra, W. Jiang, R. Manchek, and V. Sunderam, PVM 3.0 User’s Guide and Reference Manual, Oak Ridge National Laboratory, ORNL/TM-12187, February, 1993.

[75] G. A. Geist, V. S. Sunderam. Network-Based Concurrent Computing on the PVM System, Concurrency: Practice and Experience, Vol. 4(4), 293-311 (June 1992).

[76] A. Beguelin, J. Dongarra, G. A. Geist, R. Manchek, V. S. Sunderam. Graphical Development Tools for Network-Based Concurrent Supercomputing, Proc. of Supercomputing ’91, IEEE, pp. 435-445, 1991.

[77] J. Van Bladel and C. M. Butler. Aperture Problems, (Proc. NATO Adv. Study Inst. on Theoretical Methods for Determining the Interaction of Electromagnetic Waves with Structures,) Ed. by J. Skwirzynski, Sythoff and Noordhoff international Publishers, 1979.

[78] Parametric Method of Moments (ParaMoM) RCS Prediction Packages, Version 1.0, User’s Manual, Syracuse Research Corporation, TD92-1321, October, 1992.

[79] J. Bacus. Can Programming be liberated from the von Neumann style ? a functional style and its algebra of programs, CACM 21(8), pp.613-641, August 1978.

[80] A Standard for Robot Exclusion, http://info.webcrawler.com/mak/projects/robots/norobots.html.

[81] Alta Vista, http://altavista.digital.com/.

[82] InfoSeek, http://guide.infoseek.com/.

[83] Lycos, http://www.lycos.com/.

[84] Open Text, http://www.opentext.com/omw/f-omw.html.

[85] Yahoo, http://www.yahoo.com/

[86] Oracle Corp.. SQL*TextRetrieval Administrator’s Guide V2.0, July 1992.

[87] Oracle Corp.. SQL*TextRetrieval Developer’s Guide V2.0, July 1992.

[88] Oracle Corp.. Oracle Server for IBM RS/600 Scalable POWERparallel Systems Installation and Configuration Guide, Release 7.3.1 Beta, Part No. A40830-1, Dec. 1995.

[89] G. Fox, W. Furmanski, T. Haupt and S. Klasky. WebFlow: A Visual Problem Solving Environment for Wide-Area, Heterogeneous, Distributed High Performance Computing, NPAC Proposal to the NSF New Technologies Program, July 1996.

[90] G. Fox, W. Furmanski. Towards Web / Java based High Performance Distributed Computing - an Evolving Virtual Machine, IEEE Conference HPDC-5, August 1996.

[91] D. H. Crocker. Standard for the Format of ARPA Internet Text Messages, RFC 822, August 1982.

[92] B. Kantor and P. Lapsley. Network News Transfer Protocol, RFC 977, Feb. 1986.

[93] M. Horton. Standard for Interchange of USENET Messages, RFC 850, June 1983.

[94] J. Postel. Simple Mail Transfer Protocol, RFC 821, August 1982.

[95] J. Myers and M. Rose. Post Office Protocol - Version 3 RFC 1725, Nov. 1994.

[96] J. Oikarinen and D. Reed. Internet Relay Chat Protocol, RFC 1459, May 1993.

[97] J. Kramer, J. Magge and M. Sloman. Configuring Distributed Systems, Proc. of the 5th ACM SIGOPS European Workshop, September 1992.

[98] G. Cheng. ASKNPAC Internet Services at NPAC, http://asknpac.npac.syr.edu, NPAC, Syracuse Univeristy, October 1995.

[99] JavaSoft. JavaBeans — Component APIs for Java, http://splash.javasoft.com/beans/, July 1996.

[100] JavaSoft. The JDBC(tm) Database Access API, http://splash.javasoft.com/jdbc/, June 1996.

[101] JavaSoft. Remote Method Invocation and Object Serialization, http://chatsubo.javasoft.com/current/, July 1996.

[102] World Wide Web Consortium. Jigsaw Overview, http://www.w3.org/pub/WWW/Jigsaw/, July 1996.

[103] JavaSoft. Overview of the Java HTTP Server Architecture, http://www.javasoft.com/products/jeeves/CurrentRelease/doc/architecture/, June 1996.

[104] JavaSoft. Java Servlet Application Programming Interface White Paper, http://www.javasoft.com/products/jeeves/CurrentRelease/doc/api.html, August 1996.

[105] JavaSoft. The Java Language and API, http://java.sun.com:80/doc/language.html, July 1996.

[106] G. Fox el.. HPCC and Java – A Preliminary Report by The Parallel Compiler Runtime Consortium (PCRC), http://www.npac.syr.edu/users/gcf/hpjava.html, March 1996.

[107] NCSA. NCSA Habanero Project Overview, http://www.ncsa.uiuc.edu/SDG/Software/Habanero/, May 1996.

[108] G. Fox, W. Furmanski, M. Podgorny, K. Hawick and J. Niemiec, WebSpace: A WebWindows Based Gateway to the ANL LabSpace, http://www.npac.syr.edu/projects/webspace/NPAC Proposal, Syracuse University, May 1996.

[109] G. Fox, W. Furmanski, M. Chen, C. Rebbi and J. Cowie, WebWork: Integrated Programming Environment Tools for National and Grand Challenges, NPAC Technical Report, SCCS-715, Syracuse University, May 1996.

[110] JavaSoft. Remote Objects for Java, http://splash.javasoft.com/pages/intro.html, July 1996.

[111] JavaSoft. JavaTM WorkShop, http://www.sun.com/sunsoft/Developer-products/java/, March 1996.

[112] JavaSoft. Java IDL, http://splash.javasoft.com/JavaIDL/pages/index.html, May 1996.

[113] M. Baker. Cluster Computing Review, NPAC Technical Report, SCCS-748, Syracuse University, May 1996.

[114] G. Fox, W. Furmanski. Snap, Crackle, WebWindows!, NPAC Technical Report, SCCS-758, Syracuse University, May 1996.

[115] G. Laszewski. Interactive Parallel Program Generation for a Metacomputing Environmet, NPAC Technical Report, Syracuse University, August 1996.

[116] K. Chandy, A. Chelian, B. Dimitrov, H. Le, J. Mandelson, M. Richardson, A. Rifkin, P. Sivilotti, W. Tanaka, and L. Weisman, A World-Wide Distributed System Using Java and the Internet, IEEE Conference HPDC-5, Syracuse, August [117] I. Foster and S. Tuecke. Enabling Technologies for Web-Based Ubiquitous Su-

percomputing, IEEE Conference HPDC-5, Syracuse, August 1996.

[118] T. McCartney, K. Coldman, and D. Saff, EUPHORIA: End-User Construction of Direct Manipulation User Interfaces for Distributed Applications, Technical Report, Dept. of Computer Science, Washington University, August 1995.

[119] Microsoft. ActiveX, http://198.105.232.5:80/intdev/, June 1996.

[120] T. Mowbray and R, Zahavi. The Essential CORBA: Systen Integration Using Distributed Objects, John Wiley and Sons, Inc., 1996.

[121] World Wide Web Consortium. WWW and OOP, http://www.w3.org/pub/WWW/OOP/, July 1996.

[122] M. Brown. FastCGI: A High-Performance Gateway Interface, White Paper, Open Market, Inc., http://www.fastcgi.com/kit/doc/fastcgi-whitepaper/fastcgi.htm, April 1996.

[123] World Wide Web Consortium. Hypertext Transfer Protocol – Next Generation, http://www.w3.org/pub/WWW/Protocols/HTTP-NG/Overview.html, July 1996.

[124] C. A. R. Hoare. Communicating Sequential Processes, Prentice-Hall, 1985.

[125] IBM. IBM Visualization Data Explorer (DX), http://www-i.almaden.ibm.com/dx/.

[126] Khoral Research Inc. What Is Khoros?, http://www.khoros.unm.edu/khoros/khoros2/home.html.

[127] Enterprise Integration Technologies. Hypermail, http://www.eit.com/software/hypermail/.

[128] S. Hariri, G. Fox, W. Furmanski. The Virtual Distributed Computing Environment, NPAC, Syracuse University, 1996, http://merlin.cat.syr.edu/projects/vm/index.html.

Biographical Data

Name:  Gang Cheng
Date and Place of Birth:  June 1963, Sichuan, China

Degrees Awarded:

Master in Computer Engineering, 1988 Huazhong University of Science and Technology, Wuhan, China

Bachelor in Computer Engineering, 1983 Huazhong University of Science and Technology, Wuhan, China

Awards and Honors:

Supercomputing ’93 Student Paper Award, Best student paper in the area of Graphics Application Portland, Oregon, November, 1993

Co-developer of a patent issued by the People’s Republic of China, May, 1993 Professional Experience:

Research Scientist, May 1993 – Northeast Parallel Architectures Center Syracuse University, NY

Graduate Assistant, Jan. 1992 – March 1993 Northeast Parallel Architectures Center, Syracuse University, NY

Teaching Assistant, Sept. 1991 – Jan. 1992 School of Computer and Information Science Syracuse University

Programmer Analyst, April 1991 – August 1991 Information System Support Laboratory Virginia Tech, VA

Research Associate and Senior Software Engineer, August 1985 – May 1988 Chengdu Institute of Computer Applications Chinese Academy of Sciences

Application Programmer, June 1983 – July 1985 Chengdu Institute of Geography Chinese Academy of Sciences