Given by Tom Haupt at Gateway Workshop Aberdeen Maryland on 24-25 May 99. Foils prepared May 26 99
Outside Index
Summary of Material
One of four Gateway talks: Bill Asbury, Geoffrey Foox, Tom Haupt and Ken Flurchick |
This has overview of system/requirements, Middle Tier and what it means to add a component to Gateway |
Outside Index
Summary of Material
This project is a collaborative effort between
|
Tom Haupt |
NPAC Syracuse University |
111 College Place |
Syracuse |
NY 13244-4100 |
To provide a problem-oriented interface to more effectively utilize HPC resources from the desktop via the Web browser. |
This "point & click" view hides the underlying complexities and details of the HPC resources and creates a seamless interface between the user's problem description on his/her desktop system and the heterogeneous computing resources |
These HPC resources include supercomputers, mass storage systems, databases, workstation clusters, collaborative tools, and visualization servers. |
Create an illusion that all resources needed to complete the user tasks are available locally. |
In particular, an authorized user can allocate the resources she needs without explicit login to the host controlling the resources. |
An analogy: NSF mounted disk or a network printer. |
Back-End services |
comprise Tier 3. |
Distributed, object-based, |
scalable, and reusable |
Web server, Object broker |
and Resource Manager |
Middleware forms Tier 2 |
Tier 1 is a high-level, browser-based Front End |
for visual programming (including selection of applications, |
generation of input data sets, specification of resources, |
post-processing and visualizations) |
Problem description:I need to model the surface damage due to the impact |
of laser to harden the material bulk. I need access to models including |
material bulk properties and interaction with intense electromagnetic fields. |
Task description: I need 64 nodes of SP-2 at Argonne to run my |
MPI-based executable "a.out" you can find in "/tmp/users/haupt" on marylin.npac.syr.edu. In addition, I need any idle workstation with jdk1.1 installed. Make sure that the output of my a.out is |
transferred to that workstation |
Middle-Tier: map the user's task description onto the resource specification; this may include resource discovery, and other services |
Resource Specification |
Resource Allocation: run, transfer data, run |
Ken Flurchick, http://www.osc.edu/~kenf/theGateway |
1. Enter the Gateway system |
2. Define your problem |
3. Identify resources (software and hardware) |
4. Create input file |
5. Run your application |
6. Analyze results |
CTA specific knowledge databases |
Visual |
Authoring |
Tools |
User and |
Group |
Profiles |
Resource Identification and Access |
Visualizations |
Collaboration |
WebFlow |
Back-End Resources |
Problem Solving Environment |
Support for a seamless access (security) |
Support for distributed, heterogeneous Back-End services (HPCC, DBMS, Internet, ...) managed independently from Gateway |
Variable pool of resources: support for discovery and dynamical incorporation into the system |
Scalable, extensible, low-maintenance Middle Tier |
Web-based, extensible, customizable, self-adjusting to varying capacities and capabilities of clients (humans, software and hardware) front end |
Distributed, object-oriented middle tier
|
Gateway operates in a keberized environment [Support for a seamless access]
|
Task Specification is expressed in XML
|
Resource Specification is expressed in XML
|
[Support for distributed, heterogeneous Back-End services; Variable pool of resources; Scalable, extensible, low-maintenance Middle Tier] |
Component-based Front-End [extensible] |
Front-End Components ("toolbox interfaces") are
|
All components (Front End, Middle-Tier) are defined in XML and contain metadata (used for component mining) |
Subject of Ken's talk
|
Allows for composition of the computational task from components (reusable modules) |
Different tools to support various programming models such as data parallel, task parallel, data flow, object oriented |
No assumption on granularity |
Metadata about components and support for archiving and mining the components |
Support for instrumentation and steering |
Controls the user/group environment
|
Allows for customization
|
History of actions |
Scientific notebook |
Computational resources
|
Data
|
Networks |
Baker/Clarke's talk on SciVis |
Geoffrey's talk on Collaborative Tools |
support for "streaming" applications as components |
support for heterogeneous hardware (capabilities/capacities) |
Portal Page |
User Context |
Control Applet |
Navigator (extensible, customizable) |
PSE specific toolboxes
|
Other (Collaboration, Visualizations, ...) |
Provides initial access to the Gateway. |
After mutual authentication of the user and the Gateway server, creates a user context, and returns a (signed) control applet. |
Represents a Gateway session. |
The session is associated with a user (or group) profile. |
WebFlow extends the notion of the UNIX profile via the 'User Data Base' (UDB). This UDB contains information about submitted jobs, history of the users actions, and other user state information. The user context may also contain application/front-end specific information. |
The control applet is responsible for maintaining the session, and direct communication with the middle-tier. |
Direct communication is the most efficient, but since it is buried into an applet, this mechanism is not readily customizable. |
The generic services, such as file service (upload, download, edit, copy, move, delete) and job services (show current jobs/show queues/kill jobs) will be supported this way. [combination of the user context and a query] |
The Gateway will also support a non-direct communication with the middle-tier through servelts. |
The navigator allows the user to select and customize toolboxes. |
Embedded in a separate frame, it consists of menus, buttons, links, etc, derived from an XML document. |
The navigator is a hierarchical, extensible and customizable. |
The problem description is application specific, and the Gateway only provides a general framework for creating a PSE. |
The most important part is the specification of what services (middle and back tier) are needed, what is their API, and how to add new services. |
Example services: access to databases, XML parsing, generating HTML in-the-fly, file services. |
The end user see it as a mapping between the problem description and software to be used to solve the problem. Actually, it identifies WebFlow modules and their parameters to be used to construct the application (see resource request toolbox below). |
The module parameters may include input files, and if necessary, the input files are generated at this stage (using this or a separate toolbox). In addition, some parameters will be constructed from information stored in data bases, including UDB, and other sources. |
The front-end activities result in an abstract task specification. |
Abstract in the sense that the user may not know nor care what actual resources are used. |
The task is composed of independently developed modules and services following different programming models. |
Visualizations |
Collaboration |
Scientific notebook |
... |
User 1 |
User 2 |
Application 1 |
Application 2 |
App 2 |
App 1 |
WebFlow server is given |
by a hierarchy of containers |
and components |
WebFlow server hosts users and services |
Each user maintains a number of applications composed of custom modules and common services |
WebFlow Services |
Mesh of WebFlow Servers |
implemented as CORBA objects |
that manage and coordinate |
distributed computation. |
Gatekeeper |
Authentication |
Authorization |
Master Server (Gatekeeper) |
Slave Server |
Slave Server |
User Context |
Application Context |
Module |
Slave Server Proxy |
Services |
User Modules |
Browser based Front-End |
Browser |
based |
Front-End |
User Space Definition and Task Specification |
Metacomputing Services |
Back-End Resources |
Access to HPCC (via Globus) |
Access to distributed databases (via JDBC) |
Access to mass storage |
Access to the Internet resources |
Access to desktop application and local data |
Access to code repositories |
In order to run WebFlow over Globus there must be at least one WebFlow node capable of executing Globus commands, such as globusrun |
Jobs that require computational power of massively parallel computers are directed to the Globus domain, while other jobs can be launched on much more modest platforms, such as the user's desktop or even a laptop running Windows NT. |
Bridge between WebFlow and Globus |
Computational engines
|
Databases
|
SECIOP |
Front End Applet |
SECIOP |
authentication |
& authorization |
Gatekeeper |
delegation |
HPCC resources |
GSSAPI |
GSSAPI |
Layer 1: secure Web |
Layer 2: secure CORBA |
Layer 3: Secure access to resources |
Policies defined by resource owners |
. |
Gateway applications are composed of independent reusable modules |
Modules are written by module developers who have only limited knowledge of the system on which the modules will run. |
The WebFlow system hides module management and coordination functions |
Back-end service |
Middle-tier proxy |
Front-end controls |
Nothing!
|
A middle-tier proxy will submit your job for you |
Often, your job do not need to interact.
|
If you need to interact
|
Many come as a standard Gateway modules |
User's modules
|
#include "..\BC.idl" module WebFlow { |
module lms{ |
interface runEdys:BeanContextChild { void run(); void receiveData(); |
void setParameter(in string p); |
}; interface runCasc2d:BeanContextChild{ void run(); void runAgain(); |
}; interface DoneEvent{ Object getSource(); }; |
}; }; |
We will create 3 CORBA objects |
* two modules: - runEdys - runCasc2d * one event - DoneEvent They will be added to package WebFlow.lms |
May act as a client for a back-end service such as Globus GRAM or a database |
May invoke other Gateway Middle Tier services such as File Services or Resource Discovery |
May implement the desired functionality internally (say, in Java) [thus not a proxy] |
May interact with other modules and the Front End through events |
addEventListener |
rmEventListener |
fireEvent(E,M) |
method M |
Event Source |
Event Target |
Adapter |
Event |
ORB |
binding |
table |
DII |
DSI |
Details in Ken's talk |
How to control a Middle-Tier module?
|
How to build an application from modules?
|
Proxy Module |
Module |
ActionButton1 |
ActionButton2 |
.... |
IIOP |
Another complication: |
Java sandbox! |
XML |
A visual representation is converted into a XML |
document |
XML |
service |
Web |
Server |
save |
parse |
ApplContext |
Generates Java code to add modules to ApplContext |
Publishes IOR |
Front-End Applet |
Middle-Tier |