Given by Geoffrey C. Fox at MAPINT97 Workshop on 16-18 June 97. Foils prepared 27 June 97
Outside Index
Summary of Material
We distinguish multidisciplinary problems, metacomputing and Interoperable interfaces |
We need to consider difficulties of metaproblems, metacomputers and even harder meta-institutions |
we illustrate with two examples: one from Europe couples two applications (CFD and structures) |
One ASOP, from an ambitious NASA funded industry study considers Integrated Product and Process Development of a future aircraft |
We discuss in consequence of Web distributed computing framework |
Outside Index Summary of Material
Geoffrey Fox |
Syracuse University |
NPAC |
111 College Place Syracuse NY 13244 4100 |
3154432163 |
We distinguish multidisciplinary problems, metacomputing and Interoperable interfaces |
We need to consider difficulties of metaproblems, metacomputers and even harder meta-institutions |
we illustrate with two examples: one from Europe couples two applications (CFD and structures) |
One ASOP, from an ambitious NASA funded industry study considers Integrated Product and Process Development of a future aircraft |
We discuss in consequence of Web distributed computing framework |
Interoperable Interfaces: allow a given problem to be easily run on any one of many different hosts |
Multidisciplinary Problems: A single job involves multiple different modules (say CFD and structures) which can on one or more machines
|
Metacomputing: Linkage of multiple, typically heterogeneous, computers to solve a given problem -- this is often a metaproblem
|
One example is VPL -- Virtual Programming Laboratory -- interface to HPF and MPI used by Syracuse and Cornell in courses/training |
If you agree that it can be web based then naturally Implemented as Java Framework for Computing Services |
Any Job |
Computer A |
Computer B |
Computer C |
Universal Interface |
Click on SnapShots in Virtual Programming Lab |
From NIST Sp2 Web Interface by Robert Lipman http://www.nist.gov/itl/div887/sasg/websubmit/ |
From NIST Sp2 Web Interface by Robert Lipman htt/www.nist.gov/itl/div887/sasg/websubmit/ |
Metaproblem Individual Computer or Metacomputer |
Software Bus |
Computer |
Sequential or parallel or |
Hardware Bus/Network of Metacomputer |
Mapping of general Problem or metaproblem onto general local or wide area network of computers |
or |
Any Job |
Computer A |
Computer B |
Computer C |
Computer D |
Includes cases like: with a --> A and b --> B which is classic distributed computing |
As well as classic parallel computing (e.g. HPF) with a heterogeneous target |
And arbitrary mixtures thereof |
HPF Job |
12 Node SP-2 |
8 node PC Cluster |
Multidisciplinary Applications (in their simplest implementation) are distributed computing which is integration at server level in my Java Framework for distributed computing |
General Metacomputing is the much harder combination of heterogeneous decomposition and integration |
Interoperable interfaces involve setting of standards for a client server integration (Java computing services framework) |
Geoffrey Fox, Wojtek Furmanski |
Syracuse University |
NPAC |
111 College Place Syracuse NY 13244 4100 |
3154432163 |
A large number of new Java APIs for advanced Web Services are emerging from JavaSoft and partners. Many current problems such as with the NPS WebDIS networking will be soon solved in a more robust, stable and elegant fashion by the new high-level APIs. |
Java APIs are organized in Java Frameworks. Current list of frameworks include:
|
Java Security Framework - support for authentication encryption, digital signatures. |
Java Commerce Framework - Java Wallet, Java Cassettes (digital credit cards). |
Java Beans Framework - componentware API with support for GUI negotiation and merging, persistence (JAR files), event filtering, introspection, visual application builders |
Java Media Framework - Java2D (with Adobe), animation (with Macromedia), audio/video (with Intel), Java3D (with SGI), JSDA (Java Shared Data API). |
All APIs listed are either already operational within JDK1.1, or still in works (spec only or alpha or beta release) but with the final release dates in 1997. |
Numerical Computing Framework
|
Distributed Simulation Framework
|
Computing Services Framework
|
High Performance Framework
|
Distributed Computing Framework
|
These are too many frameworks and probably several are rather controversial as there is no agreed model of the use of Web in Computing
|
The most promising for early consideration are |
1) The Numerical Computing Framework which is essentially use of Java in Computational Science and Engineering |
2) The Computing Services Framework which we give a little more detail on. |
Enables development of Web Interfaces to run a given job on any computer compliant with this framework just as JDBC gives a universal interface to any relational database |
Compiling, Executing |
Scheduling jobs as in Codine or LSF |
Accessing and storing into File Systems |
Visualization Interface (how applets access server side information) |
The Computing Services Framework allows vendors to compete on User Front End (GUI) or back end services |
The framework is implemented as a set of drivers which map generic Java Interfaces to particular software (e.g. a compiler) on particular machines. |
HPJava@NPAC is a set of prototypes to explore features of a Java Framework for High Performance Computing
|
We are now integrating them together to produce a reasonably complete Java based Prototype High Performance Computing Environment |
HPCC is a small field with limited resources for a very hard problem and must leverage as much software as possible |
The Web provides an excellent pervasive user interface with Java Applets and WebWindows |
The Web provides a potentially excellent high performance object oriented language (Java) for scientific and engineering computation |
The Web provides a rich but modest performance distributed computing environment based on either Web Servers or Clients
|
So the Web provides "everything" except high performance and data parallelism(!)
|
Agree on standards (The High Performance Java Framework) so that one can maximize software re-use |
Java Wrappers (native classes or Server socket connections) around existing data parallel Fortran or C++ |
Native Java and MPI
|
Data Parallel Extensions of Java
|
Java threads for data parallelism on SMP's |
1:User View: Interoperable Web Interface accessing services through Java Compute Services Framework 2:Network of Java Servers provide distributed services to databases, compute engines, collaboratories, object brokers, instruments
|
High Performance layer accessed when distributed services have insufficient performance
|
Compute processes linked by MPI or distributed units of database, object broker, I/O system etc. |
Java Servers |
If performance adequate, use server level as universal and so certainly put all control at server level |
Coarse grain Integration at Server level |
Fine grain linkage at High Performance layer |
A |
B |
MPI |
Java Socket |
Note JDBC (Java Database Connection) links clients to servers at natural (sequential) SQL query and database connection level
|
CORBA is similar except parallel CORBA still needs to be understood!
|
Suggest Java Compute Framework similarly defines interfaces to distributed computing level
|
Compare two ways of linking two modules or two programs of a multidisciplinary application where each module has internal MPI implementation |
Server Based or High Performance MPI InterModule connectivity |
Systems like Tango or Habanero built around Java Servers integrate a group of multiple clients as a "Service" at the middle Java Server level |
Group of collaborating clients |
and client applications |
Database |
Object Broker |
MPP |
VPL -- Virtual Programming Laboratory
|
WebFlow -- dataflow programming environment at the Java Server level |
HPF Interpreter -- Java or JavaScript GUI to HPF (or MPI)
|
WebFlow gives an AVS like Interface to integrate multilanguage modules |
A C++ PCRC SPMD Physics simulation is linked with disparate modules to filter output display |
This demo is available on a video tape and with RealAudio narration |
This model can be combined with emerging Web Collaboration, Object broker and database capabilities to support multidisciplinary applications and computational steering |
User Interpreted Commands |
invoking preloaded |
Java/Javascript objects |
Server invokes extrinsic |
processor |
(true interpreter |
or compiler invoked |
dynamically) |
Running Program |
interruptable at |
extrinsic interfaces |
HPF statements and Visualization requests in Java, HPF etc. |
HPF extrinsic procedure |
Shared Object |
Lexical Analysis: PCRC frontend. |
HPF compilation: pghpf |
Pool of shared objects |
HPF RTS |
HPF |
server |
Dynamical linking |
Instrumented HPF |
JAVA |
applet |
Fortran2Java translator of Fortran code into Java language |
JavaMPI -- a partial Java Interface to MPI |
TANGOsim -- Collaboratory Technology
|
SciVis Toolkit -- Library of client side applets
|
CISPAR: Ongoing ESPRIT (European Union) project involving academia and industry from multiple countries linking two MPI codes together (chosen from example 3D fluid and 2D solid industrial codes)
|
ASOP MADIC was Syracuse led but largely industry driven and developed a full multidisciplinary multi-organization (inter) nationally distributed Integrated Product and Process Development (IPPD) framework
|
3 commercial codes (STAR-CD, PAM-CRASH, PERMAS) in CFD and structures |
End Users from Aerospace, Ship (Insurance), Automobile and Heart Valve manufacturing arenas |
High Performance computing expertise from industry (PALLAS), lab(GMD Bonn) and University (Imperial College UK) |
Duration Jan 96 to Dec 98 |
Each of two codes has steps |
Input --> Mesh Generation --> Partitioning --> MPI based simulation --> Post Processing |
CFD |
COCOLIB communication library supplies Data and Control |
Structures |
MPI Processes of individual programs |
Partition Individual Grids |
COCOLIB Interpolates between grids of 2 programs |
Need high performance linkage between two codes as couple "microscopically" at
|
In last case could perhaps use "high level" server linkage of codes |
Needs heterogeneous MPI implementations |
Designed to run with either
|
Must interpolate between different types (element shapes) of structured and unstructured meshes |
COCOLIB Choice: |
Optimal High Performance Choice |
Data and Control |
Data Only |
CFD Server |
Structures Server |
Control |
Only |
Simplest Server level Implementation |
Structures Server |
CFD Server |
A set of manufacturing companies -- Rockwell International, Northrop Grumman, McDonnell Douglas, General Electric and General Motors is studying the NII implications for a particular MAD system "Affordable Systems Optimization Process" (ASOP) |
Interesting parameters are that next major aircraft to be built could involve:
|
Application Specific NII Specific Services for
|
Critical feature is closely integrated collaboration of very many people and computers requiring |
Metacomputing Simulation and distributed database support as in Collaboratory |
Workflow support including configuration management and central CAD databases |
Rich collaboration environment to support distributed design decisions |
Standards and Security to allow interlinking of people and software from different organizations |
Key special requirement is distributed coordination |
COMMERCIAL AIRCRAFT COST REDUCTIONS ARE TARGETED AT 25% NEAR TERM |
THREE OF FIVE MILITARY FIXED-WING AIRCRAFT FAMILIES IDENTIFY REDUCING COSTS BY 20-40% (BY 2015) AS A PRIORITY TECHNOLOGY DEVELOPMENT |
Customizes general NII technologies to ASOP specific requirements |
Provides the object, wrapper and agent services to share data in the ASOP design environment |
Conforms to applicable protocol, ontological and data standards |
Defines management, retrieval, and control functions for heterogeneous data and methods distribution |
Interfaces to Team Infrastructures |
Collaboration Services - support distributed, virtual collocation of aeronautics development teams |
Configuration Control Services - provide configuration management for complex product data relationships |
Metacomputing Services - facilitate effective use of powerful, distributed computing services |
Security and Access Services - safeguard information and ensure validity of access to ASOP capabilities |
Object and Data Service - implement for data transformations and resource integration |
Employs emerging "open" NII services to support aeronautics teams and collaboration
|
Expanded for large concurrent user base
|
Access additional NII bandwidth as needed |
Focused on configuration control for aeronautics products and associated data
|
Provides services for managing common product model across distributed team
|
Supports rollback of configurations and long-term archive of product and data configurations |
Facilitates distributed "computing" services available across an ASOP project team |
Supports access to shared "supercomputer" resources on as needed basis for aeronautics ASOP projects |
Interfaces to "open" NII services for ASOP users
|
Provides range of data/system security for ASOP
|
Support access control to ASOP capabilities
|
Insure only authorized users access, modify, use ASOP services and information |
Specializes generic "object/wrapper/agent" technologies to ASOP requirements |
Provides object management; distribution; control and retrieval services |
Stores and archives ASOP related data objects for re- use |
Implements specific data exchange standards for data objects; i.e. STEP Application Protocols |