Given by Geoffrey C. Fox at Seamless Computing Conference ECMWF Reading England on Sept 17,97. Foils prepared Sept 12 97
Outside Index
Summary of Material
We discuss role of commodity (Web) technologies in future seamless computing environments |
We describe how a network of servers architecture can naturally support both parallel and distributed computing while |
We discuss implications of metacomputing, multidisciplinary applications, |
We suggest critical importance of CORBA and component based software in HPCC
|
We recommend agreement on standard interfaces or frameworks for computing |
Outside Index Summary of Material
Geoffrey Fox |
Syracuse University |
NPAC |
111 College Place Syracuse NY 13244 4100 |
3154432163 |
We discuss role of commodity (Web) technologies in future seamless computing environments |
We describe how a network of servers architecture can naturally support both parallel and distributed computing while |
We discuss implications of metacomputing, multidisciplinary applications, |
We suggest critical importance of CORBA and component based software in HPCC
|
We recommend agreement on standard interfaces or frameworks for computing |
There are a set of related concepts
|
How general should we be?
|
Can commodity (web) technologies deliver needed performance today or in near future? |
Can we agree on necessary standards/frameworks? |
Batch submission and analysis |
Proper Interactive Client Server |
General 3 Tier Architecture with multiple services
|
Client |
(Web) Server |
aka Middleware |
One example is VPL -- Virtual Programming Laboratory -- interface to HPF and MPI used by Syracuse and Cornell in courses/training |
If you agree that it can be web based then naturally Implemented as Java Framework for Computing Services |
Any Job |
Computer A |
Computer B |
Computer C |
Universal Interface |
Click on SnapShots in Virtual Programming Lab |
From NIST Sp2 Web Interface by Robert Lipman http://www.nist.gov/itl/div887/sasg/websubmit/ |
From NIST Sp2 Web Interface by Robert Lipman htt/www.nist.gov/itl/div887/sasg/websubmit/ |
Scivis is a client-server (3-tier) data visualization and analysis system by taking full advantage of Java. |
The purpose of this system is provide researchers with a customizable data analysis system to aid their research. |
We also provide a collaborative framework, where the users can exchange data and their own personalized filters. |
Available via http://kopernik.npac.syr.edu:8888/scivis |
Contains over 29K lines of Java code (2K lines of user-definable filters). |
A screen dump from a Scivis Session |
Metaproblem Individual Computer or Metacomputer |
Software Bus |
Computer |
Sequential or parallel or |
Hardware Bus/Network of Metacomputer |
Mapping of general Problem or metaproblem onto general local or wide area network of computers |
or |
Any Job |
Computer A |
Computer B |
Computer C |
Computer D |
Includes cases like: with a --> A and b --> B which is classic distributed computing |
As well as classic parallel computing (e.g. HPF) with a heterogeneous target |
And arbitrary mixtures thereof |
HPF Job |
12 Node SP-2 |
8 node PC Cluster |
Multidisciplinary Applications (in their simplest implementation) are distributed computing which is integration at server level in my Java Framework for distributed computing |
General Metacomputing is the much harder combination of heterogeneous decomposition and integration |
Interoperable interfaces involve setting of standards for a client server integration (Java computing services framework) |
One can "just" use Object Web technologies as a software infrastructure for building parallel, distributed or sequential computing environments which can have a very different architecture from the Web
|
As above, but specifically build a metacomputing environment |
One can view the Object Web as a distributed information system with modest performance and build a metacomputing system with the Web architecture
|
Harness the power of the Web as a computer -- use up the idle cycles on the WebTV's in every home -- typically a Web Client based system |
So this is partly use of Object Web Software to build metacomputing systems of any architecture |
Perhaps VRML or Java3D are important for scientific visualization |
Web (including Java applets) front-ends provide convenient customizable interoperable user interfaces to HPCC facilities |
Perhaps the public key security and digital signature infrastructure being developed for electronic commerce, could enable more powerful approaches to secure HPCC systems |
Perhaps Java will become a common scientific programming language and so effort now devoted to Fortran and C++ tools needs to be extended or shifted to Java |
The universal adoption of JDBC (Java Database Connectivity) and the growing convenience of web-linked databases could imply a growing importance of systems that link large scale commercial databases with HPCC computing resources |
Javabeans, RMI, COM, CORBA, IIOP form the basis of the emerging "pragmatic object web" which analogously to the previous bullet could encourage a growing use of modern object technology which will allow better managed distributed systems |
Emerging collaboration and other distributed information systems could allow new (low end) distributed work paradigms which can be used in computational steering and for basic scientific research |
High end (today) to high end(tomorrow)
|
High end(today) to low end(tomorrow)
|
Low end(today) to low end(tomorrow)
|
Low end(today) to very low end (tomorrow)
|
The new application(s) we can't predict |
1997 |
2007 |
Any large scale metacomputing/distributed computing environment will be shaped by and shape all 5 classes of applications on previous foil |
Even if only aimed at high end applications, the system will be influenced by and influence the "Object Web" or "commodity software infrastructure" which is here defined as "mass-market"/business IntraNet (low to low) use of Internet/distributed Information System |
Parallel Computing systems can be viewed as a special case of a Metacomputer
|
Applications are metaproblems with a mix of module and data parallelism |
Modules are decomposed into parts (data parallelism) and composed hierarchically into full applications.They can be the
|
Modules are "natural" message-parallel components of problem and tend to have less stringent latency and bandwidth requirements than those needed to link data-parallel components
|
Assume that primary goal of metacomputing system is to add to existing parallel computing environments, a higher level supporting module parallelism
|
Object Web Software provides a high functionality but modest performance distributed computing (Metacomputing) environment based on either Web (soon to be CORBA IIOP and HTTP/Java Socket) Servers or Clients |
Here we will explore an architecture using servers for control as higher functionality than clients although currently less broadly deployed
|
Object Web Only addresses Integration of already decomposed parts!
|
Middle Tier |
Basic Web Server |
Custom Web Server |
TP Server |
Business Transaction Management |
You Write Software |
at Client and Server |
Old and New Useful Backend Software |
1:User View: Interoperable Web Interface accessing services through Java Compute Services Framework |
2:Network of Java Servers provide distributed services with databases, compute engines, collaboratories, object brokers, instruments
|
Back end "Number Crunchers" linked either by communication at level 2 (slowish but easy) or at level 3 (high performance but more work) |
Compute processes linked either to servers or together by MPI if parallel |
Java Servers |
We have a set of Services hosted by Object Web Servers which form the middleware and accessed by clients |
Groups of clients (electronic societies) are linked by Java server based collaboration systems such as TANGO or Habanero |
Access |
Resources |
Store |
Multimedia Information |
Collaboration Server |
File Systems |
and/or Database |
Object Broker |
Database |
Simulation |
e.g. NEOS |
Netsolve |
Computer |
Person2 |
Shared |
WhiteBoard |
Shared Client Appl |
Person1 |
General User |
As a first step, implement multi-module systems with each module linked via Java Servers
|
Where necessary "escape" down to classic HPCC technologies for data transport keeping control at server level
|
This seems very convenient in JDK 1.1 "event model" which is mechanism used by Javabeans to communicate
|
1)Simple Server Approach 2)Classic HPCC Approach |
3)Hybrid Approach with control at server and |
data transfer at |
HPCC level |
4)Invoke High Performance Message Transfer between Observers and Sources specified in Message Event |
Server Tier |
Data Source |
Data Sink (Observers) |
5)Actual Data Transfer |
High Performance Tier |
2)Prepare |
Message Event in Source Control |
1)Register Observers with Listener |
Here are some examples of using our approach where large scale industry investment in Web technology appears to add significant value to metacomputing systems built with Web architecture
|
Multidisciplinary and Computational Steering Applications
|
Visual and Interpreted Programming Environments
|
Technologies to get High Performance CORBA |
Integration with Forces Modeling (Distributed Event driven Simulation) |
Integration with Networked enabled servers such as NEOS and Netsolve
|
Simulation |
Basic Display |
Image Filter |
is another |
module |
Output Display after Filter |
Runs as a |
parallel |
module |
using |
Java Server |
host |
Bunch of Filters and Displays |
defined in |
Java Graph editor and |
running on grid of Java Servers |
Original Image |
Visual Basic/C++/J++ and ActiveX or Beanboxes with Javabeans give visual approach to software objects
|
Enterprise Javabeans and COM are extending this to distributed computing |
Using Web technologies for grid and building modules out of (whatever Javabeans/COM evolves to) allows one to deliver to user HPCC programming environments with comparable friendliness to those in PC world |
They are Java's implementation of "component-based" visual programming |
This modern software engineering technique produces a new approach to libraries which become a "software component infrastructure(SCI)" |
There is a visual interface to discovery of and setting of values of and information about parameters used in a particular software component |
JavaBeans uses the event model of JDK1.1 to communicate between components
|
The visual interface allows inspection of and implementation of both individual beans and their linkage . This visual construction of linkage allows one to form nontrivial programs with multiple communicating components |
Apart from the event mechanism which is a communication/linkage mechanism, ComponentWare (and JavaBeans in particular) "just" give a set of universal rules (needed for interoperability) for rather uncontroversial (albeit good) object-oriented and visual programming practices
|
In general it is any process, but it is convenient (in the pure form of our web approach) to view each module as a Javabean (or equivalent component) |
The Javabean can wrap existing Fortran, Perl or C C++ code by either using native methods or by invoking the code as a separate process |
Modules as Javabeans allow them to be stored as objects and inspected visually
|
Wrapping existing code as Javabeans is a good way of renovating "legacy code" so can be used more easily in future!
|
Large gains in HPCC user productivity will be attained if we can integrate the ideas and technologies of modern (PC) visual programming with classical HPCC approaches |
Use of important emerging Web and CORBA technology allows HPCC object (C++.,Java) and visual (CODE, Hence, WebFlow, AVS, Khoros) systems to be enhanced to become parallel component-based visual programming systems. |
CORBA does not incorporate HPCC but as it specifies services and not implementation,
|
HP-CORBA can be built on Nexus and Globus and it will allow HPCC users access to any CORBA service with an option for high performance when necessary. |
The NPAC WebFlow technology can be combined with emerging JavaBean technology to produce a prototype HPcomponent system. |
Note industry is ahead with sequential ComponentWare but is only now moving with activeX to distributed systems. HPCC already has visual distributed environments. So HPCC need not be behind if it generalizes modules to Javabeans |
We present the proposed process of integrating HPCC and Enterprise Computing technologies on 2-dimensional 3x3 chart, exposing the evolution of user and system perspectives on new computing technologies. |
On the (vertical) system/complexity axis, we start from PC desktop and we evolve towards distributed and finally HPCC computing. |
On the (horizontal) user/simplicity axis, we start from objects and we evolve towards reusable components and finally their visual development and runtime environments. |
The first row on our char represents the already established PC technologies exemplified by systems such as Visual C++/J++, VBScript, Borland Delphi, Visual Cafe etc. |
The second row corresponds to the emergent Object Web based Enterprise Computing that integrates Java/JavaBeans based component technologies with CORBA based distributed objects. |
Finally, the third row represents our proposed HPCC extensions of these technologies in terms of HP-CORBA based HP-Components. |
At present, JavaBeans technology is focused on component programming within a single Java VM. |
Within the Sun philosophy of '100% Java', distributed Beans can be developed using RMI interconnect. However, the rest of the industry tries to protect their C++ investments while converting to Java. |
Hence, in parallel with JavaBeans development, the Web industry explores now the linkage of Java with CORBA based distributed object technologies which offer an full C++/Java interoperability. |
CORBA supports cross-language remote object invocation as well as IIOP (Internet Inter-ORB Protocol) based interoperability between object brokers from various vendors. |
Of particular interest are Java based ORBs or ORBlets which can be downloaded as applets to enable CORBA capabilities also at the client/browser side. |
An alternative, offered by Netscape, is a resident ORB support in all browser and server products. |
Java based ORBs will soon turn the Web, so far acting as a largely passive document publishing framework, into a powerful dynamic world-wide distributed object-based computing environment. |
Now consider how to make the Object Web High performance. |
In the object domain we propose the extension HP-CORBA of the CORBA model for the HPCC domain by developing a minimal high performance ORB on top of MPI/Nexus. |
Such HP-ORBlets, residing in the individual nodes of a parallel system (either as network daemons or as runtime libraries) would allow one to hide explicit message passing calls in terms of higher level more user-friendly remote object invocations.
|
HP-ORBlets would focus on high performance data/method communication support, whereas the lower bandwidth control communication would be passed to and handled by the full functionality lower performance commercial ORBs in the CORBA bus in the middleware layer. |
The split between data and control communication would be fully transparent at the program's level, i.e. both parallel object developers and integrators would be offered a uniform CORBA object based programming model.
|
Two major advantages of the proposed approach are:
|
A large number of new Java APIs for advanced Web Services are emerging from JavaSoft and partners. Many current problems such as with the NPS WebDIS networking will be soon solved in a more robust, stable and elegant fashion by the new high-level APIs. |
Java APIs are organized in Java Frameworks. Current list of frameworks include:
|
Java Security Framework - support for authentication encryption, digital signatures. |
Java Commerce Framework - Java Wallet, Java Cassettes (digital credit cards). |
Java Beans Framework - componentware API with support for GUI negotiation and merging, persistence (JAR files), event filtering, introspection, visual application builders |
Java Media Framework - Java2D (with Adobe), animation (with Macromedia), audio/video (with Intel), Java3D (with SGI), JSDA (Java Shared Data API). |
All APIs listed are either already operational within JDK1.1, or still in works (spec only or alpha or beta release) but with the final release dates in 1997. |
Numerical Computing Framework
|
Computing Services Framework
|
High Performance Framework
|
Distributed Computing Framework
|
Distributed Simulation Framework
|
Java Calls (mainly Interfaces and not methods) to capabilities expressed in implementation neutral form |
Drivers convert these general calls to vendor specific implementation of service |
Java code can either be all on client (2-tier) or on client and middle tier (3 tier) |
e.g. JDBC is a universal interface to all relational databases |
Adoption of this JDBC implies that vendor specific solutions (such as Oracle's PL/SQL) are immediately less attractive
|
Note database business is larger than simulation business
|
These are too many suggested computing frameworks and probably several are rather controversial as there is no agreed model of the use of Java in Computing
|
The most promising for early consideration are |
1) The Numerical Computing Framework which is essentially use of Java in Computational Science and Engineering |
Very important to get agreement on areas that affect JavaVM and the Java Language as these are getting harder to change
|
2) The Computing Services Framework which we give more detail on. |
Enables development of Web Interfaces to run a given job on any computer compliant with this framework just as JDBC gives a universal interface to any relational database
|
The Computing Services Framework allows vendors to compete on either User Front End (GUI) or back end services with the JavaCS framework providing universal linkage |
The framework is implemented at the backend as a set of drivers which map generic Java Interfaces to particular software (e.g. a compiler) on particular machines. |
Requires agreement by "suitable interested parties" on
|
http://www.sis.port.ac.uk/~mab/Computing-FrameWork/ |
Is CORBA (viewing system as a collection of objects) useful |
Compiling, Executing, Specification of features needed for execution optimization
|
Accounting -- use Web commerce technology? |
Security (especially hard in metacomputing as link several different management policies)
|
Sharing, Accessing and Storing into File Systems |
Data and Performance Visualization Interface (how applets access server side information) |
Performance measurement and recording (cf: Pablo SDDF) |
Interfaces for Programming Tools
|
Libraries including names in Math class and
|
Module linkage model for metaproblems (multidisciplinary applications) as in Javabeans sufficient? |