Given by Tom Haupt at CEWES Training on WebFlow at Vicksburg on March 9 1999. Foils prepared March 26 1999
Outside Index
Summary of Material
Erol Akarsu (*) |
Geoffrey Fox |
Tomasz Haupt |
Alexey Kalinichenko (*) |
Kang-Seok Kim (*) |
Praveen Sheethalnath (*) |
Choon-Han Youn |
(*) student |
Synergistic projects |
(led by W. Furmanski) |
at NPAC: FMS |
Object WebHLA |
JWORB http://bombay.npac.syr.edu/fms |
also Tango |
Outside Index Summary of Material
presented by |
Tomasz Haupt Northeast Parallel Architectures Center at Syracuse University |
This project is sponsored by the U.S. Army Corps of Engineers Waterways Experimental Station MSRC (Vicksburg, MS) under the DoD Modernization Program, Programming Environment and Training |
Erol Akarsu (*) |
Geoffrey Fox |
Tomasz Haupt |
Alexey Kalinichenko (*) |
Kang-Seok Kim (*) |
Praveen Sheethalnath (*) |
Choon-Han Youn |
(*) student |
Synergistic projects |
(led by W. Furmanski) |
at NPAC: FMS |
Object WebHLA |
JWORB http://bombay.npac.syr.edu/fms |
also Tango |
Part I: Introduction (25 min) |
Part II: WebFlow Design (45 min) |
Part III: WebFlow Security (20 min) |
10:30 -10:45 break |
Part IV: WebFlow Applications (75 min) |
12:00-1:30 lunch |
Part V: How to Write WebFlow modules (15 min) |
Part VI: LMS details, Demos, Discussion |
Introduction |
seamless access to remote resources
|
high-level user friendly visual programming and runtime environment for HPDC |
portable system based on industry standards and commodity software components |
Remote Resources |
FRONT-END: |
high-level user friendly |
- visual programming and authoring tools - application GUI |
RESOURCES: |
all hardware and software |
components needed to complete the user task, including, but not limited to, |
compute engines from |
workstations to supercomputers, |
storage, databases, instruments, |
codes, libraries, and licenses. |
Desktop/Laptop |
Seamless |
Access |
Create an illusion that all resources needed to complete the user tasks are available locally. |
In particular, an authorized user can allocate the resources she needs without explicit login to the host controlling the resources. |
An analogy: NSF mounted disk or a network printer. |
WebSubmit (NIST) |
TeraWeb (NCS, Inc.) |
CCM PSE (OSC) |
many others |
Web Browser |
SP-2 |
O2K |
CGI |
CGI |
SSL |
Disadvantage: |
- client/server based on custom protocol over CGI |
Advantages: |
- platform independent mini-language (RSL) for specification of resources |
- can be layered on top of different schedulers - enables interoperability between resources (can allocate many resources at a time, file transfer, monitoring, etc.) Disadvantage: |
- a bag of low level tools |
PSE: problem description (physics, chemistry, ...) |
Task description: I need 64 nodes of SP-2 at Argonne to run my |
MPI-based executable "a.out" you can find in "/tmp/users/haupt" on marylin.npac.syr.edu. In addition, I need any idle workstation with jdk1.1 installed. Make sure that the output of my a.out is |
transferred to that workstation |
Middle-Tier: map the user's task description onto the resource specification; this may include resource discovery, and other services |
Resource Specification |
Resource Allocation: run, transfer data, run |
Remote Resources |
We need a third tier! |
Problem |
Solving |
Environments |
OO Visual |
Authoring |
Tools |
Data-Flow |
Visual |
Authoring |
Custom |
Application |
GUI |
Other |
WebFlow |
Back-End Resources |
PSE |
OO |
Data Flow |
Custom |
GUI |
Other |
WebFlow |
DATORR, Alliance |
DATORR, Alliance |
HPCC: Globus |
Other as needed |
DBMS: JDBC |
Small tasks: Java |
user codes |
https, IIOP/SECIOP |
Example: IPSE |
Under development |
NCSA Alliance |
Example: LMS |
Navigate and choose an existing application to solve the problem at hand. Import all necessary data. |
Retrieve data |
Pre/post-processing |
Run simulations |
Select host |
Select model |
Set parameters |
Run |
Ken Flurchick, http://www.osc.edu/~kenf/Gateway |
1 .Define your problem |
2. Identify resources (software and hardware) |
3. Create input file |
4. Run your application |
5. Analyze results |
Compose interactively |
your application |
from pre-existing |
modules |
Data-Flow Front-End |
WebFlow Design |
Follow industry standards
|
Participate in creation of standards
|
Use commodity components
|
Development of WebFlow coincides with DATORR standardization initiative |
Could lead to definition of CORBA facilities or Java framework for computing services |
Defines:
|
http://www-fp.mcs.anl.gov/~gregor/datorr/ soon to become http://www.datorr.org |
Object Oriented, follows JavaBeans model
|
Firing event E by object A causes invocation of method M of object B. The association of event E and method M is achieved by an event registration mechanism. An event is also an object and it carries data. |
(a digression) |
more information on Java, Corba, Distributed Object: |
http://www.npac.syr.edu/projects/cps616spring96/index.html |
Typically WebFlow objects live in different address spaces. We use CORBA to invoke methods of the remote objects. |
- Objects A and B are CORBA objects (thus not Java objects) |
- Objects are defined in IDL (Interface Definition Language) |
- IDL definitions are compiled using (Java)IDL compiler |
- The IDL compiler generates new classes to be used by the Java compiler |
(javac) instead of the original ones, on both the client and server side |
- The IDL compiler generates either classes to be extended, or interfaces to be implemented |
Object Adapter |
serves also as a daemon |
#include "..\BC.idl" module WebFlow { |
module lms{ |
interface runEdys:BeanContextChild { void run(); void receiveData(); |
void setParameter(in string p); |
}; interface runCasc2d:BeanContextChild{ void run(); void runAgain(); |
}; interface DoneEvent{ Object getSource(); }; |
}; }; |
We will create 3 CORBA objects |
* two modules: - runEdys - runCasc2d * one event - DoneEvent They will be added to package WebFlow.lms |
WebFlow objects are developed independently of each other (reusable modules): we cannot assume that the event source knows anything about the event target and vice versa |
addEventListener |
rmEventListener |
fireEvent(E,M) |
method M |
Event Source |
Event Target |
Adapter |
Event |
ORB |
binding |
table |
DII |
DSI |
Proxy Module |
Module |
ActionButton1 |
ActionButton2 |
.... |
IIOP |
Another complication: |
Java sandbox! |
Local Host |
Add module |
Module Factory |
Proxy Module |
Remote Host |
FE |
request |
Add module |
Module Factory |
Module |
The WebFlow server is a container object, a.k.a. context - in fact it implements JavaBeanContext class (Java1.2) |
The BeanContext acts as a logical container for JavaBeans ("WebFlow modules and services") and BeanContexts. |
User 1 |
User 2 |
Application 1 |
Application 2 |
App 2 |
App 1 |
WebFlow server is given |
by a hierarchy of containers |
and components |
WebFlow server hosts users and services |
Each user maintains a number of applications composed of custom modules and common services |
WebFlow Services |
Mesh of WebFlow Servers |
implemented as CORBA objects |
that manage and coordinate |
distributed computation. |
Gatekeeper |
Authentication |
Authorization |
Master Server (Gatekeeper) |
Slave Server |
Slave Server |
User Context |
Application Context |
Module |
Slave Server Proxy |
Services |
User Modules |
Data Flow |
Front-End |
OO |
Front-End |
User Space Definition and Task Specification |
Metacomputing Services |
Back-End Resources |
Similar to JavaBeans
|
May serve as Proxies
|
Services are modules provided by the system and offers a generic functionality
|
The Run Job module is a proxy module. It generates the RSL on-the-fly |
and submits the job for execution using globusrun function. |
The module knows only exec name, location and its arguments/parameters. |
In order to run WebFlow over Globus there must be at least one WebFlow node capable of executing Globus commands, such as globusrun |
Jobs that require computational power of massively parallel computers are directed to the Globus domain, while other jobs can be launched on much more modest platforms, such as the user's desktop or even a laptop running Windows NT. |
Bridge between WebFlow and Globus |
WebFlow Security |
(design) |
Access Control (or Authorization)
|
Authentication
|
Integrity
|
Accountability (or non-repudiation)
|
Privacy
|
Mutual authentication of servers and users
|
Access control
|
Privacy |
Integrity |
SECIOP |
Front End Applet |
https |
authentication |
& authorization |
Gatekeeper |
delegation |
Stakeholders |
HPCC resources |
GSSAPI |
GSSAPI |
Layer 1: secure Web |
Layer 2: secure CORBA |
Layer 3: Secure access to resources |
Policies defined by resource owners |
https (SSL) |
AKENTI |
CORBA security service |
GSSAPI (Globus) |
can play both client and server
|
evolve continually
|
interactions are not well defined
|
are polymorphic (ideal for Trojan horses!) |
can scale without limit
|
are very dynamic |
Secure Communications |
Authentication |
Client |
User |
Encryption |
Audit |
Authorization |
Server |
Encryption |
Credentials |
Object |
Adapter |
ORB |
A principal is authenticated once by ORB and given a set of credentials, including one or more roles, privileges, and an authenticated ID. |
An authenticated ID is automatically propagated by a secure ORB; it is part of the caller context |
Principal |
Credentials |
Current |
Client |
Server |
set_credentials |
get_attributes |
authenticate |
No delegation
|
Simple delegation
|
Composite delegation
|
Client |
Target |
Client |
Target |
Client |
Target |
Client |
Target |
Object |
IIOP |
Based on a trusted ORB model: you must trust that your ORB will enforce the access policy on the server resource |
The ORB determines: if this client on behalf of this principal can do this operation on this object |
Server uses Access Control Lists (ACL) to control user access |
Principal |
Role |
Rights |
Operation |
Mary Thompson, http://www-itg.lbl.gov/security/Akenti/DOE2000/sld014.htm |
WebFlow Applications |
Applications vary by the functionality of their Front-Ends
|
Applications vary by how they are composed from modules
|
The modules can interact with each other in different ways:
|
Applications vary on how the Front-End interacts with the Middle-Tier
|
To develop a web based system that implements a "navigate-and-choose" paradigm and allows the end user to:
|
Anytime, anywhere, using any platform |
(e.g., a connected to the Internet laptop PC)
|
A decision maker (the end user of the system) wants to evaluate changes in vegetation in a geographical region over a long time period caused by short term disturbances such as a fire or human activity. |
One of the critical parameters of the vegetation model (EDYS) is soil condition at the time of the disturbance. |
This in turn is dominated by rainfall that possibly occurs at that time (CASC2D simulation) |
Input data for the simulations are available from the Internet, such as Data Elevation Models (DEM) from USGS web site or from custom databases (spices characteristics) |
Data retrieval |
Data preprocessing |
Simulation: two interacting codes
|
Visualization |
WMS |
EDYS |
CASC2D |
DEM |
Land Use |
Soil |
Texture |
Vegetation |
EDYS: vegetation model |
CASC2D: watershed model |
WMS: Watershed Modeling System |
Data retrieval |
Data pre- and post-processing |
Simulations |
The data wizard allows the user to interactively select the data and |
download them to the local machine. The raw data are then fed to the WMS system launched from the browser to generate input files for simulations. |
Launching coupled simulations on different Back-End computational resources |
The results of the |
simulations are send |
back to the Front-End, |
and can be visualized |
using tools included |
in WMS package |
Front-End (client) is a Java application
|
"navigate and choose" - no interactive composition of applications
|
modules exchange data through message passing mediated by WebFlow |
client keeps the module references |
slave |
runCasc2d |
master |
UNIX |
WinNT |
slave |
runEdys |
lms.class |
Data wizard |
WMS |
exeCasc2d |
Web |
Server |
Web |
Server |
WebFlow Servers |
Client |
- WebFlow modules |
Start web servers on both machines |
Start master on WinNT |
Start slave on WinNT |
Start slave on UNIX |
Start client (Java lms) on WinNT |
try { |
//add modules |
p1 = slaveNT.addNewModule("runEdys"); //as defined in conf.file |
runEdys re = runEdysHelper.narrow(p1); |
p2 = slaveUNIX.addNewModule("runCasc2d"); //as defined in conf.file |
runCasc2d rc = runCasc2dHelper.narrow(p2); |
//bind events |
master.attachEvent(p2,"Casc2dDone","Casc2dDone",p1,"run"); |
master.attachEvent(p1,"EdysStarted","EdysStarted",p2,"run"); |
master.attachEvent(p1,"EdysDone","EdysDone",p2,"runAgain"); |
//invoke methods of runCasc2dImp |
rc.run(); |
} |
catch(COMM_FAILURE ex) |
{System.err.println(ex.getMessage()); System.exit(1);} |
slave |
runCasc2d |
master |
UNIX |
WinNT |
slave |
runEdys |
lms.class |
Data wizard |
WMS |
exeCasc2d |
Web |
Server |
Web |
Server |
slave |
runCasc2d |
master |
UNIX |
WinNT |
slave |
runEdys |
lms.class |
Data wizard |
WMS |
exeCasc2d |
Web |
Server |
Web |
Server |
casc2d |
slave |
runCasc2d |
master |
UNIX |
WinNT |
slave |
runEdys |
lms.class |
Data wizard |
WMS |
exeCasc2d |
Web |
Server |
Web |
Server |
casc2d |
slave |
runCasc2d |
master |
UNIX |
WinNT |
slave |
runEdys |
lms.class |
Data wizard |
WMS |
exeCasc2d |
Web |
Server |
Web |
Server |
casc2d |
slave |
runCasc2d |
master |
UNIX |
WinNT |
slave |
runEdys |
lms.class |
Data wizard |
WMS |
exeCasc2d |
Web |
Server |
Web |
Server |
casc2d |
slave |
runCasc2d |
master |
UNIX |
WinNT |
slave |
runEdys |
lms.class |
Data wizard |
WMS |
exeCasc2d |
Web |
Server |
Web |
Server |
casc2d |
slave |
runCasc2d |
master |
UNIX |
WinNT |
slave |
runEdys |
lms.class |
Data wizard |
WMS |
exeCasc2d |
Web |
Server |
Web |
Server |
casc2d |
slave |
runCasc2d |
master |
UNIX |
WinNT |
slave |
runEdys |
lms.class |
Data wizard |
WMS |
exeCasc2d |
Web |
Server |
Web |
Server |
casc2d |
Write |
slave |
runCasc2d |
master |
UNIX |
WinNT |
slave |
runEdys |
lms.class |
Data wizard |
WMS |
exeCasc2d |
Web |
Server |
Web |
Server |
casc2d |
IIOP |
http |
Write |
http |
slave |
runCasc2d |
master |
UNIX |
WinNT |
slave |
runEdys |
lms.class |
Data wizard |
WMS |
exeCasc2d |
Web |
Server |
Web |
Server |
slave |
runCasc2d |
master |
UNIX |
WinNT |
slave |
runEdys |
lms.class |
Data wizard |
WMS |
exeCasc2d |
Web |
Server |
Web |
Server |
Front-End (client) is a Java applet |
applications are created dynamically from pre-existing modules |
modules exchange data through ports (data flow model) |
server keeps the module references; the references are published on a web site |
XML |
A visual representation is converted into a XML |
document |
XML |
service |
Web |
Server |
save |
parse |
ApplContext |
Generates Java code to add modules to ApplContext |
Publishes IOR |
Front-End Applet |
Middle-Tier |
<!DOCTYPE taskspec [ |
<!ELEMENT taskspec (task)+> |
<!ATTLIST taskspec |
UserContextRef CDATA #REQUIRED |
AppName CDATA #REQUIRED> |
<!ELEMENT task ((task | module)*,connection*) > |
<!ELEMENT module (#PCDATA) > |
<!ATTLIST module |
modulename CDATA #REQUIRED |
host CDATA #REQUIRED > |
<!ELEMENT connection (out,in)> |
<!ELEMENT in EMPTY> |
<!ELEMENT out EMPTY> |
<!ATTLIST out |
modulename CDATA #REQUIRED |
eventname CDATA #REQUIRED |
<!ATTLIST in |
modulename CDATA #REQUIRED |
method CDATA #REQUIRED > |
]> |
<taskspec UserContextRef="123as321" AppName="TestApplication"> |
<task> |
<module modulename="FileBrowser" host="localhost"> |
</module> |
<module modulename="FileEditor" host="localhost"> |
</module> |
<module modulename="Gaussian" host="localhost"> |
</module> |
<connection> |
<out modulename="FileBrowser" eventname="FileEvent" event="File"/> |
<in modulename="FileEditor" method="run"/> |
</connection> |
<connection> |
<out modulename="FileEditor" eventname="FileEvent" event="File"/> |
<in modulename="Gaussian" method="run"/> |
</connection> |
</task> |
</taskspec> |
Coordinates transformations |
Remote HPCC resources |
- Web interface to store data in DB in variable format - Data transfer from DB to a visualization engine |
- Coordinates transformations on |
a remote server - Launching simulations on remote |
hosts with interactive input |
Applet |
Application |
Context |
Netscape ORB |
ORBacus ORB |
IIOP |
List of servers |
List of modules |
List of events |
List of methods |
E |
M |
Add module |
Attach Event |
local |
remote |
Adapter |
LLM |
Back-End services |
comprise Tier 3. |
Tier 1 is a high-level Front-End for visual programming |
Distributed object-based, scalable, and |
reusable Web server and Object broker |
Middleware forms Tier 2 |
Master Server is started by administrator
|
Slave Server is started by administrator
|
User Context is created by Servlet
|
Application Context is created by User
|
Module are added by User
|
Portal |
Page |
Secure |
Web Server |
Mutual |
authentication |
start |
AKENTI |
Credentials |
Globus Cert. |
Front End |
Applet |
WebFlow |
Server |
User |
Context |
Netscape's ORB |
ORBacus ORB |
IIOP |
. |
WebFlow applications are composed of independent reusable modules |
Modules are written by module developers who have only limited knowledge of the system on which the modules will run. |
The WebFlow system hides module management and coordination functions |
Single Web-based access via Gateway portal |
Security based on standards: https, PKI, secure ORB, GSSAPI (SSL/Keberos5) |
Access policies controlled by stakeholders |
WebFlow API allows implementation of many different front-ends |
Modern three-tier architecture (distributed objects) |
Access to HPCC through metacomputing services |
A production version is being developed within Gateway project (ASC/OSC)
|
A preliminary version is available now |
I am looking for WebFlow applications and I am ready for a collaboration |
Middle-Tier
|
Web Server (optional)
|
Secure Middle-Tier: ORBacusSSL, Akenti |
Recommended HPCC backend: Globus or Globus GSS-API |
Front-Ends (from simple to sophisticated) |
Contact person: Tomasz Haupt |
haupt@npac.syr.edu |
voice (315) 443-2087 |
http://www.npac.syr.edu/users/haupt/WebFlow/demo.html |
#include "..\BC.idl" module WebFlow { |
module lms{ |
interface runEdys:BeanContextChild { void run(); }; interface runCasc2d:BeanContextChild{ void run(); }; interface DoneEvent{ Object getSource(); }; |
}; }; |
We will create 3 CORBA objects |
* two modules: - runEdys - runCasc2d * one event - DoneEvent They will be added to package WebFlow.lms |
#include "..\BC.idl" module WebFlow { |
module lms{ |
interface runEdys:BeanContextChild { void run(); } interface runCasc2d:BeanContextChild{ void run(); } interface DoneEvent{ Object getSource(); }; |
}; }; |
The compiler will create 3 Java interfaces: - runEdysOperations.java - runCasc2dOperations.java - DoneEventOperations.java You have to implement these interfaces. Without --tie option the compiler will generate classes to be extended. We already extend from BeanContextChild |
module WebFlow { |
module lms{ |
interface runEdys:BeanContextChild { void run(); }; |
package WebFlow.lms import WebFlow.*; import WebFlow.event.*; |
import org.omg.CORBA.*; |
public class runEdysImpl extends WebFlow.BeanContextChildSupport implements runEdysOperations { String msg; public runEdysImpl(org.omg.CORBA.Object peer, String msg) { super(peer); this.msg=msg; } public void run() { ... } } |
Everything means: (for runEdys):
|
master.conf |
Server name = master |
File=D:\Jigsaw\Jigsaw\WWW\Gateway\IOR\master.ref URL=none |
Modules:================================== runEdys lms.idl WebFlow.lms.runEdysImpl runCasc2d lms.idl WebFlow.lms.runCasc2dImpl Events:=================================== EdysDone ..\BC.idl WebFlow.lms.EdysDoneImpl IDL:WebFlow/lms/EdysDone:1.0 |
Casc2dDone ..\BC.idl WebFlow.lms.Casc2dDoneImpl IDL:WebFlow/lms/Casc2dDone:1.0 |
slave.conf |
Server name = master |
File=none URL= http://kasia.npac.syr.edu:8001/Gateway/IOR/master.txt |
Modules:================================== runEdys lms.idl WebFlow.lms.runEdysImpl runCasc2d lms.idl WebFlow.lms.runCasc2dImpl Events:=================================== EdysDone ..\BC.idl WebFlow.lms.EdysDoneImpl IDL:WebFlow/lms/EdysDone:1.0 |
Casc2dDone ..\BC.idl WebFlow.lms.Casc2dDoneImpl IDL:WebFlow/lms/Casc2dDone:1.0 |
Public class Client { |
public static void main(String args[]) { |
configReader config; |
ORB orb = ORB.init(args, new java.util.Properties()); |
String masterURL = args[0]; |
String ref=getIORFromURL(masterURL); |
org.omg.CORBA.Object obj=orb.string_to_object(ref); |
WebFlowContext master=WebFlowContextHelper.narrow(obj); |
WebFlowContext slave; |
try { |
org.omg.CORBA.Object p1,p2; |
slave=WebFlowContextHelper.narrow(master.getWFServer("ntserver")); |
--------> make sure that the name match slave.conf |
p1 = slave.addNewModule("runEdys"); p2 = slave.addNewModule("runCasc2d"); |
waitForData(){ |
waitForUpdate=true; |
while (waitForUpdate) { |
idle for 1 sec |
newMod = (new File(testFile)).lastModified; if(newMod>lastMod) waitForUpdate=false; } |
sendData(); fireEvent("Casc2dDone",ev); } |
sendData(){ createContents [OutContents] copy files from casc2Dir to OutFileBase |
} |
Casc2dDone |
event |
Class cas2dThread extends Thread{ |
run(){ |
Process p=Runtime.getRuntime().exec(Casc2dExec); p.waitFor(); }} |
run(){ |
cT = new cas2c2Thread(); cT.start(); waitForData(); } |
runAgain(){ |
receiveData(); moreEvents =nextEvent(lmsStatusFile); |
lastMod=(new File(testFile)).lastModified(); if(moreEvents) { |
reactivateCasc2d(touchCommand); waitForData(); } } |
receiveData(){ |
getHTTPfile(ContentsFile,ContentsFileURL); for i=0;i<nfiles; i++){ |
getHTTPfile(casc2dDir+fn, FileBaseURL+fn);} }} |
EdysDone |
event |
called from Front-End |
receiveData(){ |
receiveStat(param3,param4,edysend); |
receiveEDY(); |
run(); } |
receiveEDY(){ |
getHTTPfile(ContentsFile,ContentsFileURL); if(ContentsFile.equals("end")) flag=false; else { |
for i=0;i<nfiles; i++){ ... translate names *.edy -> edys expectations |
getHTTPfile(EdysInDir+fn, FileBaseURL+fn);} } |
receiveStat(int, int, long){ |
readHTTPfile(StatFileURL); |
... StartDay, DayDiff ...options writeFile(OptionsFile,options); //options.txt |
run(){ |
Process p = Runtime.getRuntime().exec(EdysExec); |
p.waitFor(); if(flag) { sendData(); fireEvent("EdysDone",ev);} } |
sendData(){ createContents [OutContents] copy files from EdysOutDir to OutFileBase |
} |
Casc2dDone |
event |
EdysDone |
event |
Contact person: Tomasz Haupt |
haupt@npac.syr.edu |
voice (315) 443-2087 |
http://www.npac.syr.edu/users/haupt/WebFlow/demo.html |