Replied: Tue, 09 Mar 1999 10:07:54 -0500 Replied: "Gregor von Laszewski" Received: from antares.mcs.anl.gov (mcs.anl.gov [140.221.9.6]) by postoffice.npac.syr.edu (8.9.3/8.9.3) with SMTP id CAA22624 for ; Tue, 9 Mar 1999 02:24:57 -0500 (EST) Received: from beauty (slip127.slip.anl.gov [146.137.250.28]) by antares.mcs.anl.gov (8.6.10/8.6.10) with SMTP id BAA25653; Tue, 9 Mar 1999 01:24:02 -0600 From: "Gregor von Laszewski" To: "Dennis Gannon" , "Ian Foster" , "Geoffrey Fox" , "Piyush Mehrotra" , "Gregor von Laszewski" Subject: Datorr: position paper Date: Tue, 9 Mar 1999 01:24:30 -0600 Message-ID: <000101be69fd$df7239c0$1cfa8992@beauty> MIME-Version: 1.0 X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0002_01BE69CB.94D7C9C0" Content-Length: 31057 This is a multi-part message in MIME format. ------=_NextPart_000_0002_01BE69CB.94D7C9C0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit I include the first draft of the datorr position paper to this e-mail in html. You can also retrieve the information from the WWW under http://www.mcs.anl.gov/~gregor follow to the Datorr link and go to Position statement. It would be nice to receive your comments on this draft as they include some of my own opinions which might not conform with everyone in the list. This is also th ereason why I have not posted the draft to the mailing list. First I would like to make sure that we agree upon a common base. Parts which are not filled in are marked with [Task: ] I also could not find an official statement about the Gateway project, though I located on an NPAC machine the proposal. Since I am not sure if this is the actual mission statement, it would be nice to get a one paragraph summary. Gregor ------=_NextPart_000_0002_01BE69CB.94D7C9C0 Content-Type: text/html; name="datorr-position.html" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="datorr-position.html" Datorr

 

Datorr: Desktop access to  remote resources
            &= nbsp; Position Statement (Draft)

Gregor von Laszewski, Mathematics and Computer Science Division, = Argonne National Laboratory, Argonne, IL 60439
gregor@mcs.anl.gov, http://www.mcs.anl.gov/~gregor

based on discussions with various contributors.

[Geoffrey C. Fox, Ian Foster, Piyush Mehotra, Dennis Gannon, many = more to add, ...]

[Task: I need your approval to place your name here.]

Status 7 March, 1999.

Abstract

This paper describes the definition of the "Desktop access to = remote resources (Datorr)" project. The Datorr project is a colaborative to define a = standard which allows the interoperability of services provided by projects in the area = of metacomputing and distributed computing. The standard is based on a general = architecture which is open and extendable. Datorr is in th eprocess of defining a prototype = implementation which demonstrates the standrad by interfacing multiple services provided by = different projects with each other.

Contents

1. Introduction and Summary
2. Activities
2.1     Outreach
2.1.1         Current = Impact
2.1.2         Publicity
2.2.2         Current and = Future Outreach Activities
2.2     Research = Activities
3.0 Prototype
3.1     Prototype = Utilizing Remote Resources
3.1.1         Process Implementation
3.1.2         What Does the Prototype Leave Out?
3.1.3         Early Success Possibilities
3.2    Prototype Utilizing = Tasks
4. Future
5. References
A. Related and Complementary = Projects

1. Introduction and Summary

Today's emerging technologies in the fields of metacomputing and = distributed computing provide a substantial toolbase for building and utilizing a large scale = compute infrastructure. Improvements on the low level and high level = infrastructure are under way by a variety of projects.  Nevertheless, it is still problematic to = gain access to these resources or utilize a variety of services provided by different = projects from the users common desktop environment. Interoperability between the projects = is often problematic since this issue was previously not considered in the design = of the next generation of software.

In order to build the next generation software tool, it is desirable = to harvest this potential of development capabilities while reusing each others = technologies. Datorr strives to make use of this potential while initiating a collaborative = effort between the projects to support a standard  which can be adapted and supported = by a large number of the various projects.

In order to outline the goal of the project more precisely, it is = important to first specify the terminology used.

The term "remote resource" is defined to include:=20

  • compute-resources and = communication-resources:
    which include supercomputers, networked supercomputers and = metacomputers, network of workstations, and workstations, ...
  • data-resources:
    which includes databases, directories, dataservices, ..., and
  • compute-services and communication-services:
    which include specialized problem solving and visualization = environments, ... .

The term "desktop" is defined to includes the following = components:=20

  • desktop-hardware-device:
    a desktop-hardware-device is used to connect to the remote = resources. This includes workstations, Laptops, PDA's, and can potentially include low end = devices like pagers.
  • desktop-software:
    specifies the software running on the desktop. This includes = operating systems (UNIX/Linux, NT, ...) as well as software tools which is already = installed on remote resources (Globus, Legion, ..., CORBA, EXCEL, ..., Electronic = Notebook)
  • desktop-infrastructure:
    the term desktop-infrastructure encompasses both, = desktop-hardware-devices and desktop-software.

The term "access" is defined to include=20

  • methods of accessing remote resources including API's, protocols, = and modes of operation.   

A goal of Datorr is to come to an agreement with a variety of = projects to design a standard to enable the  interoperability between existing and = future technologies. This standard is to be defined with the input of the international = community. The goal of Datorr is not to competing  with current projects but to = collaborate. Ideas contributed by other projects should be considered while shaping such a = standard, thus the participation towards a standard is open to the community. This = ambitious goal can be started with intermediate steps as defined in the upcoming section about = "Research Contributions."

Part of the "Research Contributions" is to define a = mechanism which allows to formulate such a standard for interoperability for a variety of = services. To demonstrate the applicability and usefulness of this standard it will be necessary = to defining a prototype implementation. This prototype implementation must allow to=20

  • demonstrate the interoperability, while utilizing technologies = provided by multiple projects
  • demonstrate that the interoperability is based on the Datorr = technology for formulating the standard
  • demonstrate that the standard definition is easy to understand and = implement on a variety of platforms/frameworks.

We suggest to base the definition of the standard on an object = specification based on the extended markup language (XML) or derivatives of it. Tools should be = provided which allow the automatic integration of objects based on this = definition.These tools can include either object-translators or access mechanisms to the objects by = the service embracing the standard.

We expect that Datorr will have a mutual development effect amongst = the various projects in the community. Already today we see such an effect arising = since a common agreement between a variety of projects and research projects and = organizations exists that such an effort is highly desirable.

The current activities of Datorr include a set of international = Workshops, the development of a standard, and the development of a prototype = architecture and reference implementation.Datorr has been active since November 1998 and has been = officially introduced during SC98 in its own BoF session as well as through the = JavaGrande Forum.

2. Activities

The activities Datorr are concentrating currently on outreach, the = definition of a process to participate, and research activities.

2.1 Outreach

Datorr's intention is to bring the research community together and to = initiate dialogue and cooperation in the areas of interest. We view it as important to = include also the international community as to achieve larger impact. Since many projects = in this area are started and there is at present no central place for collecting pointers = to these projects, it is important that Datorr maintains a collection of = information related to current projects which might be useful to consider for advances in the = area of interest. Datorr could become a major part of the Gridforum and vice versa. Datorr = would allow to enhance the outreach of the Gridforum into the l international = community. Datorr would gain from the technical expertise available through the Gridforum in the = areas of security as well as Information Discovery. This symbiosis of the two efforts = would provide a beneficial advantage for the development of a standard.

2.1.1 Current Impact

Datorr has presently been successful in reaching out to the = international research community. In previous workshops participants of major international = research projects have participated. In particular, representatives of the following = projects have participated and are aware of the Datorr workshops:

Globus, Legion, Ninf, Ninja, Unicore, Webflow, Websubmit, ... = [Task: put all the other projects in here] 

2.1.2 Publicity

In order to help exchanging ideas, a mailing list = datorr@mcs.anl.gov, and datorr-announce@mcs.anl.gov are established. The datorr mailinglist is used for general discussions between the = datorr members, and the datorr-announce mailing list is used for general announcements. A WWW is = available (http://www-fp.mcs.anl.g= ov/~gregor/datorr) that contains information related to the Datorr Effort, as well as a = collection of resources including the online proceedings of the workshop (currently, a = set of slides presented at the meeting). The WWW page is expected to be changed to www.datorr.org in near future. The = WWW page is hosted by Argonne National Laboratory.

2.1.3 Current and future outreach activities

The most recent Datorr activities were delayed due = to spawning activities from other projects in similar areas. The next meeting is = planned either for June or July 1999.

2.2 Research Activities

The research activities of Datorr currently focus on the definition = of a standard architecture to incorporate desktop access to remote resources. The = component sof this architecture is defined by resources and services utilizing the = resources. Policies will describe who these resources can be accessed when and by whom. The = definition of protocols will describe how they can be accessed. An information service which = allows to access the state of the infrastructure must be central part of the architecture. = The architecture must be able to support a variety of base infrastructure including IIOP, = CORBA, Directory Services, RMI through JavaBeans, DCOM, and others.

An additional research activity should focus on the maintenance and = distribution of the software developed by the participating institutions. It would be = desirable that an institution can subscribe to an update service, which allows to upgrade = outdated   software on a particular machine. This is analog to the desire to = discover changes in the hardware structure by, for example, upgrading the memory of a = workstation. Both tasks should be automated as much as possible in order to reduce the = maintenance cost of the infrastructure. Due to the lack of current funding in this area by the = funding organizations this task could become a major contribution of Datorr = activities.

Performance issues while "plugging" infrastructure together = might play an important role while designing a standard acceptable by the high = performance community. The research effort should address and evaluate the performance while = using this approach and provide solutions which will be sufficient for the user community = addressed. Such performance requirements can be different for various communities and = the system architecture preferred by a user should be derived in such a flexible = way that it is adapted for the particular requirements.

3.0 Prototype

We suggest to develop a prototype implementation. In this = implementation, the standard is formulated on the basis of a number of  XML objects. Each = service describes a set of objects which are manipulated by the service. Services with the same = object specifications can communicate with each other and can exchange their = results similar to a dataflow model.

The first problem in defining a prototype is to identify a subset of = objects to be included in a standard. One suggestion is to develop a prototype which = is able to submit to batch queuing systems which manage a number of supercomputers, as = well as to interactive sessions. To describe such a model we require a model to = represent the memory, as well as the collection of computing nodes. Lessons from the Globus = project must be considered which allow to specify a dual view of the resource with = details or as an an abstract compute resource without detailed knowledge. Dependent on the = object definition, a query mechanism has to be defined. Naturally, the standard must be = adapted by or ported to existing major metacomputing infrastructure components, like Condor, = Globus, and Legion.

The Globus project has currently identified that it is possible to = derive such an object representations based on the Metacomputing directory service = which will be adopted as standard mechanism by NASA IPG, NPACI, ALLIANCE, and others. The = Globus project suggests to build a layer on top of the existing MDS infrastructure in = order to leverage between the LDAP object specification and the XML object specification. = This extended MDS will allow to utilize XML tools which will become the defacto standard = for publishing information and objects in the internet. This allows to utilize the = enormous software tools which we expect will be developed for PC based desktops based on = WindowsNT and UNIX/Linux.

In summary the following steps have to be followed in order to com to = a successful prototype:=20

  • Define XML definitions of a subset of resources according to = previous principles
  • Complete an implement quickly (estimated time one month)
  • Test or evaluate it (estimated time one month)

The test and evaluation should include =20

    • A verification of user and system builder requirements (by DoE, = DoD, International Projects, etc.)
    • Condor, Globus, Legion, and Unicore, and others, should be = consulted if the proposed resource definition would work
    • The Discovery and  lookup, as well as, the  use of the = resources should be documented.

This process should be iterated in order to extend the subset to = other resources.

DATORR 1.0 would be released at SC99 with the requirement that the = framework is extensible. At the same time a demo of its use should be = demonstrated.

3.1 Prototype Utilizing Remote Resources

The following section addresses issues related to remote resources, = while section 3.2 will address a prototype implementation based on task interactions.

3.1.1 Remote Resource

We suggest to build a prototype definition of compute resources based = on the following subset of resources, which represent a variety of state of the are = supercomputers:=20

    • IBM SP
    • NOW/CPLANT
    • Tera
    • Sun E10000
    • Origin 2000
    • T3E

This list includes multiprocessor nodes and SMP’s, the node = linkage between the processing elements should be expressed. Due to the complexity of such = systems this effort should be coordinated with the help of the system administrative = contacts at the centers of DOE/DoD/NPACI/ALLIANCE. 

A performance service should be defined, which takes the current = machine characteristics and issues a query to the TOP500 Supercomputing list in = order  to acquire performance data for the machines. A query should be formulated = which selects a machine based on the "linpack" performance characteristics. = Convenient mechanisms to register a compute resource with the Lookup service must = be defined. AN example for such a registration mechanism is to register the resource = via  HTTP. In addition the Globus team recommends to also provide a mechanism to = register via LDAP.  

Browsing capabilities should be developed in order to support the = hierarchical, and multiple view of compute resources.

3.1.2 Implementation Process

The implementation has to be performed by a variety of people. We = assume we need the following fulltime personal=20

  • 2 full time researcher to define and implement the prototype = definition,
  • a field test should be prepared and completed by many different = institutions,
  • 2 full time researchers will be needed to clarifying the general = principles
  • The general principles activity includes building XML base = infrastructure supporting extensibility, multiple views, hierarchy

A special emphasize has to be placed on what other projects have = learned and teach us. Issues include=20

    • Overt: syntax implies
    • Covert: what they learnt but didn’t write down

A first action should include a "letter" to Globus, Legion, = Condor, Unicore, Ninf, Ninja, Netsolve, and others which report on this issues. If = possible, this letter should also address the issue of discussing the tradeoffs to enable = useful/easy interfaces via Globus etc.

An issue to be addressed is to study and implementation a simple = datorr interface and its "managerial level" web display. The experience gained by = the Globus project can be utilized (opportunity for NCSA?)

3.2 Prototype utilizing Task interactions

One of the important features of Datorr is the fact of a task = abstraction definition.   A task is a static representation of "what" is = to be done. A Job is an instantiation of a task for execution. A task has a core set of = attributes, is extensible, and can be recursively defined. Based on the discussions of = the 2nd Datorr workshop we define a simple task or basic task object = contains:=20

  • a set of resources:
    • name, type, constraints, …
  • a set of constraints - an arbitrary expression using resource = attributes which returns a ternerary value (true, false, undefined); these define = cross resource constraints
  • a preference/rank - an arbitrary expression using resource = attributes which provides a ranking for resources matching the constraints
  • other task attributes:
    • set of inputs/outputs
    • triggers
    • action
    • executable
    • command line arguments
    • environment variables/contexts
    • owner
    • ...

 A task object contains:=20

  • a set of resources:
    • name, type, constraints, …
  • a set of subtasks
  • a set of constraints - cross resource/task constraints
  • a preference/rank
  • a set of dependencies:
    • temporal
    • data dependency
    • concurrency
    • asynchronous (absence of dependency)
    • control
  • other task attributes

 These preliminary definitions require a verification be the = community. In order to clarify this definition, existing projects have to be further = analyzed by the current concept of a task. For the prototype implementation the focus is placed = on a a simple task to start a remote process. Thus it is important to complete the = definition of a simple task for a remote process. The XML definition can be derived straight = forward form it. The resulting prototype should be commented by the community  and the = process of refining the prototype should be iterated. For a demonstration we suggest to = build a simple task matching service, which allows to execute a combination of tasks on a = dynamically selected set of resources.

3.3 What Does the Prototype Leave Out?

At present the prototype leaves out issues related to:=20

  • the access method, the security, the scheduling including the = current load
  • storage, database an other non compute resources
  • access mechanism as opposed to lookup/registration mechanism
  • a full range of www.datorr.org services like, scaling, fault tolerance, security etc. research issues

3.4 Early Success Possibilities

Early success possibilities would be a leverage and symbiosis with = Legion, Globus,   Condor, and  ... . This could be achieved while providing = complementary lookup and discovery services which are currently not provided. It would be a = success if it is possible to download an object specification or interface of a remote = resource which allow the location of additional services based on this interface. It would be = a success if a resource which is part of the exiting Grid infrastructure would be = interfacing with a desktop component which is currently not supported by the Grid. It would = be a success if through the technology suggested here such resources could be added to = the Grid infrastructure.

We expect that the reference implementation and the standard = definition will lead to publications at workshops and conferences.

4. Future

The future of Datorr seems to be promising as we are of the opinion = that it does not compete with current activities but allows to extend the current = technologies with valuable additions. This is not only based on the research component as = suggested in the previous section, but also with the the possibility of the outreach = towards national and international projects.

5. References

[GridForum] The Grid forum WWW page. http://www.gridforum.org
[XMIwww] = http://www.ontogenics.com/repository/TranslatorIntroduction.htm
[XMIpaper] Dave Carlson, Component Interoperability with XML, = Component Strategies, October 1998, Ontogenics Corp. Column: Objects & The Web ht= tp://www.ontogenics.com/research/papers/ObjectMag/Oct98.htm
[NPACgateway] Interoperable Problem Solving Environment (IPSE) = for ASC http://tapetus.npac.syr.edu/hpcmp/fms/pet/doc/gateway/Gateway4_.doc
[CCAF] Common Component Architecture Forum http://z.ca.sandia.gov/~cca-f= orum/
[ScientificIDL] Scientific IDL http://www.llnl.gvo/csnn(?)/ba= bel

Appendix

A. Related or Complementary Efforts

It is important to point out that Datorr has not the same goals as = the "Gird Forum", the DOE2000 Schema working group, the alliance interface = group, the Gateway project, and the common component architecture. We have included mission = statements of the projects to support this issue.

Datorr maintains its unique characteristic since it will provide some = unique characteristics which are so far not delivered by any one project.

The definition of a standard with international input is a unique = effort.

A.1 Grid Forum: 

The Grid Forum is an informal consortium of institutions and = individuals working on wide area computing and computational grids: the technologies that = underlay such activities as the Alliance's National Technology Grid, NPACI's = Metasystems efforts, NASA's Information Power Grid, DOE ASCI's DISCOM program, and other activities = worldwide [GridForum].

Two activities are currently being undertaken in the context of the = Grid Forum:=20

  • Grid Information Service: Definition and development of a = Lightweight Directory Access Protocol (LDAP)-based information service for representing resources = (networks, computers, etc.), people, computations, and other entities of interest in Grid = environments.
  • Grid Security Infrastructure: Definition and development of a = Public Key Infrastructure (PKI)-based system for Grid systems in which users require = single-sign-on capabilities for resources distributed over multiple administrative domains.

A.2 DOE2000 Schema Working group:

The DOE2000 effort is task to define a common LDAP based schema for = objects that are being used across DOE2000 applications.

The list of objects to be represented include:=20

  • people
  • resources: machines, software, instruments 
  • sessions/rooms
  • data

The overall goal of this effort is to promote=20

  • cooperation and interoperability between DOE2000 projects by = identifying commonalties in object definitions,
  • developing an LDAP schema for these objects, and promoting the use = of directory services for storing these objects.

An additional focus is to define representations for data objects = that would improve the ability=20

  • to discover/re-discover data after several years, e.g. by = including a globally unique identifier.

At present it seams that this effort is parallel to the = Gridforum/Alliance effort to define a production-quality LDAP server. We expect that the definition = of the obejects produced by this group will be included in the "Grid" = specifications.

Alliance: 

  • Interface Working group [Task: Describe]
  • Post Web Architecture [Task: Describe]

Gateway Project: 

See [NPACgateway].

Common Component Architecture Forum:

The mission of the Common Component Architecture Forum is to develop = a specification for a component architecture for high-performance computing. The CCA's = intent is to provide a pan-DOE lab framework that would provide a plug and play = interface for the DOE's many high performance numerical components and tools. The CCA is not = restricted to DOE, however and universities and commercial labs are welcome to participate. = Further, it is our intent that these components can be mixed and matched to create = high-performance applications on the fastest and highest-capacity machines on the planet. = Examples of these components are equation solvers, explicit stencil solvers, and attendent = load-balancers; everything that is needed to compose a high-performance simulation and = make it go. [CCAF]

 

------=_NextPart_000_0002_01BE69CB.94D7C9C0--