Full HTML for

Basic foilset Overview of DATORR and Resource working Group

Given by Geoffrey C. Fox at Second DATORR Meeting Sandia National Laboratory on February 15-16 1999. Foils prepared February 20 1999
Outside Index Summary of Material

Second Meeting of DATORR Group
Desktop Access to Remote Resources
This Presentation contains Motivating and Overview Remarks at Start of Meeting
Followed by Summary of Working Group on Remote Resources

Table of Contents for full HTML of Overview of DATORR and Resource working Group

 Denote Foils where Image Critical
Denote Foils where HTML is sufficient
1 Desktop Access to Remote Resources
2 Logistics
3 Motivation for Datorr Why desktop access?
4 Seamless Interfaces
5 What is Scope of Datorr
6 Some Cosmic Issues
7 Datorr Process
8 Workshop Activities: Details of Datorr Process
9 Contents of Proceedings
10 What Should we do this meeting?
11 Notes taken during meeting of working group on What is a remote resource (back-end view)
12 Sources of Remote Resource Taxonomies
13 Resources
14 Choices and Issues I
15 Choices and Issues II
16 Choices and Issues III
17 Choices and Issues IV
18 Choices and Issues V
19 Choices and Issues VI
20 Choices and Issues VII
21 www.datorr.org I
22 www.datorr.org II
23 Issues in prototyping XML definition of resource subset
24 Process Guidelines
25 Agreed Remote Resource Prototype
26 Process Implementation
27 Process Implementation II
28 Process Implementation III
29 What Prototype leaves Out
30 Early Success Possibilities

Outside Index Summary of Material

HTML version of Basic Foils prepared February 20 1999

Foil 1 Desktop Access to Remote Resources

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
An International HPCC/HPCN Community Activity to establish Interoperability Standards
Second Working Group Meeting Sandia February 15,16 99
http://www-fp.mcs.anl.gov/~gregor/datorr
datorr@mcs.anl.gov
HTML version of Basic Foils prepared February 20 1999

Foil 2 Logistics

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
Thanks to Sandia -- Judy Beiriger, ...
and to Argonne -- Gregor von Laszewski
This presentation consists of two merged documents
Firstly the overview given at start of meeting and
Secondly the material prepared during discussion and wrap up of "Remote Resources" working group
HTML version of Basic Foils prepared February 20 1999

Foil 3 Motivation for Datorr Why desktop access?

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
Very many machines, platforms, Operating Systems
Though a good/necessary thing, using them in changing environment becomes difficult for users.
Goal:
  • Enable Uniform or seamless access to general backend compute, communication and information resources
  • Hide details and satisfy "majority of" customers
Datorr will help metacomputing but goals are focussed on a subset of issues and aimed at establishing interoperability standards -- reference implementations may follow but are not direct goal
HTML version of Basic Foils prepared February 20 1999

Foil 4 Seamless Interfaces

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
Seamless Interface
Database1
Database2
User View
System View
2 working groups at
last meeting
3rd working groups was
a particular service -- security
HTML version of Basic Foils prepared February 20 1999

Foil 5 What is Scope of Datorr

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
Desktop
  • Workstation UNIX/MS, WWW Browsers
  • PalmPilot, Pager :-)
Which submit "tasks" to:
Remote Resources in multi-tier systems
  • compute/communication resources:
    • supercomputers, workstations, PC's
  • Data Resources:
    • databases, information services
  • compute/communication services
HTML version of Basic Foils prepared February 20 1999

Foil 6 Some Cosmic Issues

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
We are roughly defining properties and methods of those distributed objects associated with computing services -- these include jobs and resources
This could lead to CORBA facilities or Java frameworks for computing services which was motivation for interest by Java Grande Forum at www.javagrande.org
At a recent meeting at Argonne organized by NCSA Alliance, Datorr appeared to be very relevant to building reusable "Problem Solving Environment" tools a.k.a. components for domain specific "workbenchs"
HTML version of Basic Foils prepared February 20 1999

Foil 7 Datorr Process

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
Identification and documentation of existing relevant projects
Set of meetings leading to agreement on common standards
International community input such as SC98 BoF
Preliminary documents (on web) leading to (draft) final proposed standards
  • Target date for draft proposal is SC99 (another BoF) where we can have appropriate broad discussion
  • Note Metacomputing Project Symposium in Juelich, Germany Sept 20-21 1999
Selected community projects e.g. reference implementations
HTML version of Basic Foils prepared February 20 1999

Foil 8 Workshop Activities: Details of Datorr Process

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
What is the current state?
  • What projects are out there?
  • What are their goals?
  • What can we reuse?
Where do we have to expand/focus?
Most importantly workshop participants agreed that it was indeed useful to define common interoperable interfaces
Mailing List datorr@mcs.anl.gov
http://www-fp.mcs.anl.gov/~gregor/datorr
Proceedings of first workshop available
HTML version of Basic Foils prepared February 20 1999

Foil 9 Contents of Proceedings

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
32 (15 more to add!) Existing Projects surveyed covering metacomputing, distributed scheduling, seamless interface, particular tools and technologies in areas such as resource management/monitoring/collaboration/security
  • Has grown from 18 at SC98 -- add your project!
http://www-fp.mcs.anl.gov/~gregor/datorr
http://www-fp.mcs.anl.gov/~gregor/datorr/report/datorr-report.html
http://www-fp.mcs.anl.gov/~gregor/datorr/report/datorr-security.html
HTML version of Basic Foils prepared February 20 1999

Foil 10 What Should we do this meeting?

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
First we should have an update from nifty projects which were not discussed at the first meeting
Then we should break into working groups in areas which aren't too controversial so we lay a solid foundation for future Datorr activities
Suggested topics:
  • What is a task (user view)
  • What is a remote resource (back-end view)
  • What are services (linking users and resources)
  • Architecture Issues
HTML version of Basic Foils prepared February 20 1999

Foil 11 Notes taken during meeting of working group on What is a remote resource (back-end view)

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
HTML version of Basic Foils prepared February 20 1999

Foil 12 Sources of Remote Resource Taxonomies

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
Globus
Condor
Legion
Netsolve, Ninf
GRD (Genias/Raytheon)
NOW Millenium (UCB)
Ninja (as services)
Jini
DoD Gateway, DRM (in planning phase)
Dutch ASCI (Bal)
Harness
TOP500 and other Linpack registration and publication services
PetaSIM and other performance estimators (Jim Browne, Warwick, Saltz at Maryland)
HTML version of Basic Foils prepared February 20 1999

Foil 13 Resources

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
CPUs
System Architecture
Databases
Software -- libraries, licenses, applications, versions
Network QoS, Bandwidth
Storage
  • Parallel I/O
Visualization
Memory
Event Handlers
Peripherals such as instruments
Printers
Cost: Money (accounting) and User Pleasure
HTML version of Basic Foils prepared February 20 1999

Foil 14 Choices and Issues I

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
Extensibility
Resource management (temporal versus spatial)
  • What are criteria of successful scheduling
  • Millenium is user preference oriented
  • Interactive / Computational Steering
  • Availability
Services imply resources and vice versa
Queue Length depends on dynamic issues
HTML version of Basic Foils prepared February 20 1999

Foil 15 Choices and Issues II

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
Express user needs, preferences, bank balance
Language to compose resources into constraints etc.
  • Condor is not very relatively unique?
  • Mariposa is a database system
  • Jini has limited capabilities
  • even Globus has it!
Look up and discovery
Security characteristics are difficult to express
How to express standard?
  • C++ API is not essential
  • CORBA or Java RMI Interfaces
  • Web Interfaces
  • Cell Phone (except in DoE facilities)
Separate control path (discover resource) from data path (how you use)
Standard should specify nature of API
HTML version of Basic Foils prepared February 20 1999

Foil 16 Choices and Issues III

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
So access method is part of resource definition
  • cf. Javelin from UCSB
High Level Principles?
  • Platform Independence -- required at least for resource discovery
  • Linkages to common approaches
  • Separate resource discovery from resource use
  • Define protocols that resources respect
Hierarchical definitions
is user's bank balance a resource?
HTML version of Basic Foils prepared February 20 1999

Foil 17 Choices and Issues IV

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
A Legion is a resource
  • Globus views Legion as a software resource
  • Legion views Globus .....
Note Globus could refuse to let you ask what machines where in its resource pool
What is architecture of a resource
  • all resources have a natural service attached to them -- a printer is a printing service exporting the printer interface?
  • Need strong typing of interface
Sometimes discovery is trivial -- you remember your money is stored under your bed etc.
What resources will the real world define
  • Some we should ignore (e.g. cars)
  • Others we should follow the big boys (e.g. printers?)
HTML version of Basic Foils prepared February 20 1999

Foil 18 Choices and Issues V

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
What is on our list
  • Either because we can't describe them well
  • Or we don't need them
  • We should focus -- decide on limited goals
Discovery is not performance critical -- can use standards even if non optimal
How does one query resource list
Need both required and optional arguments
Should document why certain things are there e.g. maybe some fields only there for some special computers
HTML version of Basic Foils prepared February 20 1999

Foil 19 Choices and Issues VI

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
Look into relationships with MIME types
A given resource will have multiple XML descriptions depending on type of query
  • e.g. some queries need just high level descriptions of MPP; others detailed hierarchical definitions
So need to specify types of query to define what one needs to define a resource
Grand Challenge in XML is (courtesy of NPACI/UCB) distinguish Tera T90 old SP new SP, cluster of PC's WS's
HTML version of Basic Foils prepared February 20 1999

Foil 20 Choices and Issues VII

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
Ninja's list of interesting resources is broader but less detailed than Datorr
Do we support multiple network interface cards reflecting different uses of system
Identify successes and failures of previous systems
HTML version of Basic Foils prepared February 20 1999

Foil 21 www.datorr.org I

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
A unifying concept is www.datorr.org which web site will support registration , lookup and display of world's compute resources
  • If you can do lookup, you must be able to do registration
Scope of www.datorr.org is "computing" as the commodity market is not addressing
  • cf. HLA standard in DoD
www.datorr.org appears a more modern "better" model than Globus Legion etc.
  • hotpage from SDSC is a prototype of this web site?
  • As hotpage supports Globus, if we can link Globus to www.datorr.org, we will get hotpage!
HTML version of Basic Foils prepared February 20 1999

Foil 22 www.datorr.org II

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
We should build into the initial prototype a "generic" or "simple" interface which is a low level API by which remote resource returns to www.datorr.org (perhaps in system dependent XML which could be later standardized) status information
We should build an elegant web display of the information returned by the "generic" interface
HTML version of Basic Foils prepared February 20 1999

Foil 23 Issues in prototyping XML definition of resource subset

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
What is the subset? One suggestion is
  • Submit to a batch queues managing N supercomputers
  • Submit interactively
After we decided that this is too hard -- do this later
As part of this discussion, we discussed issues such as:
  • Do supercomputers require memory model and notion of collection of nodes?
  • Or are they an abstract compute resource without detailed knowledge of internal structure
  • They are of course both!
What is query mechanism?
  • Access www.datorr.org with XML tools ....? Record world's supercomputers (TOP500)
  • Map into Legion, Globus, or Condor syntax
HTML version of Basic Foils prepared February 20 1999

Foil 24 Process Guidelines

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
Goal: Need XML definition of resources according to previous principles
Choose a subset -- implement quickly e.g. one month
Test or evaluate it (also a month)
  • User and system builder requirements from DoE DoD etc.
  • Globus Unicore Legion Condor Ninja etc. can ask if this proposed resource definition would work
  • Cover lookup and use of resource
Iterate process extending subset to other resources
DATORR 1.0 at SC99 does not preclude DATORR 1.1 etc. later (i.e. must be extensible)
SC99 demo of its use at www.datorr.org should include task definition (the other working group)
HTML version of Basic Foils prepared February 20 1999

Foil 25 Agreed Remote Resource Prototype

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
Specify a given collection of computers: subset of
  • IBM SP
  • NOW/CPLANT i.e. cluster of workstations and PC's
  • Tera
  • Sun E10000
  • Origin 2000
  • T3E
include multiprocessor nodes (include digital SMP's) and node linkage
Query XML database for "linpack" performance
Must build on the XML base infrastructure supporting extensibility, multiple views, hierarchy
Registration service to add resources to www.datorr.org comes together with lookup service
HTML version of Basic Foils prepared February 20 1999

Foil 26 Process Implementation

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
Identify "people" with responsibilities
  • Prototype definition (small team)
  • Preparation of tests(many),
  • Clarifying general principles (small team)
An important initial activity is building XML base infrastructure supporting extensibility, multiple views, hierarchy
HTML version of Basic Foils prepared February 20 1999

Foil 27 Process Implementation II

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
What other projects teach us
  • Overt: syntax implies
  • Covert: what they learnt but didn't write down
  • "letter" to Globus Legion Condor Unicore Ninf Ninja Netsolve Performance projects -- response by end of march
XML Architecture
  • UCB see if can do by end of march
www.datorr.org design principles
  • UCB and ... (search, scaling services/principles)
www.datorr.org hosting and implementation
  • ANL and .... (could need work on scaling etc.)
Project Description
  • NPAC/ANL
HTML version of Basic Foils prepared February 20 1999

Foil 28 Process Implementation III

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
XML specifications of prototype (Around April 1 start)
  • NPAC ANL
  • discuss tradeoffs to enable useful/easy interfaces via Globus etc.
Testing and Evaluation of both XML and Website (also April 1)
  • Distributed DoD center at Maui, NCSA ,DoD MSRC at ASC, DoE
  • maybe SDSC
  • linking Globus
Study and implementation of "generic" datorr interface and its "managerial level" web display (later than April 1)
  • TBD -- ANL has experience (e.g. Java Globus resource)
  • NCSA interested
HTML version of Basic Foils prepared February 20 1999

Foil 29 What Prototype leaves Out

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
Access method, security, scheduling including current load
storage, database an other non compute resources
Access mechanism as opposed to lookup/registration mechanism
Full range of www.datorr.org services
  • scaling, fault tolerance, security etc. research issues
  • Ninja certainly investigating many of these
HTML version of Basic Foils prepared February 20 1999

Foil 30 Early Success Possibilities

From Overview of DATORR and Resource working Group Second DATORR Meeting Sandia National Laboratory -- February 15-16 1999. *
Full HTML Index
Legion Globus Condor ... use www.datorr.org
Download interface to remote resource with query
Legion Globus etc. supply different interfaces to given remote resources
Use of "generic" www.datorr.org interface to get machine status by working scientists
© Northeast Parallel Architectures Center, Syracuse University, npac@npac.syr.edu

If you have any comments about this server, send e-mail to webmaster@npac.syr.edu.

Page produced by wwwfoil on Sat Feb 20 1999