Full HTML for

Basic foilset MetaComputing -- the Informal Supercomputer

Given by Mark Baker at Tutorial for CRPC Annual Meeting at Argonne on May 13 1996. Foils prepared August 4 1996
Outside Index Summary of Material

Distributed Computing.
The Challenge.
Understanding the Functionality of a Metacomputer.
Workings of Typical Cluster Management Software.
Features of Metacomputing Management Software:
  • - Environments and application support.- Job scheduling and allocation policy.- Configurability and resource dynamics.
Status of CMS Packages - basic problems.
Some Current Metacomputing Projects:
  • - Legion - UVa.- Wide Area Metacomputer Manager (WAMM) - Italy.- Wide Area Network Environment (WANE) - FSU.
Near and Future Projects:
  • - WWW/CGI - RSA Factoring.- JAVA based systems - MetaWeb.
Metacomputing in the future !

Table of Contents for full HTML of MetaComputing -- the Informal Supercomputer

 Denote Foils where Image Critical
Denote Foils where HTML is sufficient
1 Metacomputing: The Informal Supercomputer
2 Overview
3 Overview
4 Overview
5 Alternative Supercomputing Resources
6 Parallel/Distributed Computing - Communications Characteristics
7 Some Comments about Parallel and Distributed Computing
8 Communications Performance of Some Parallel and Distributed Systems
9 Distributed Systems: Some Problems
10 Distributed Computing Systems
11 Management Software for Distributed Systems
12 Management Software for Distributed Systems: The Problem
13 Distributed Programming Paradigms
14 The Challenge
15 The Challenge
16 Understanding the Functionality of a Metacomputer
17 Existing Cluster Management Software
18 Existing Cluster Management Software
19 Some Currently Available CMS Packages
20 Some Currently Available CMS Packages
21 The Workings of Typical Cluster Management Software - 1
22 The Workings of Typical Cluster Management Software - 2
23 The Workings of Typical Cluster Management Software - 3
24 The Workings of Typical Cluster Management Software - 4
25 The Workings of Typical Cluster Management Software - 5
26 The Workings of Typical Cluster Management Software
27 Desirable Features of Metacomputing Management Software
28 Desirable Features of Metacomputing Management Software
29 Desirable Features of Metacomputing Management Software
30 Desirable Features of Metacomputing Management Software
31 Desirable Features of Metacomputing Management Software
32 Desirable Features of Metacomputing Management Software
33 Desirable Features of Metacomputing Management Software
34 Desirable Features of Metacomputing Management Software
35 Desirable Features of Metacomputing Management Software
36 Desirable Features of Metacomputing Management Software
37 Desirable Features of Metacomputing Management Software
38 Desirable Features of Metacomputing Management Software
39 Desirable Features of Metacomputing Management Software
40 Desirable Features of Metacomputing Management Software
41 Desirable Features of Metacomputing Management Software
42 Desirable Features of Metacomputing Management Software
43 Desirable Features of Metacomputing Management Software
44 Desirable Features of Metacomputing Management Software
45 Desirable Features of Metacomputing Management Software
46 Summary of Desirable Metacomputing Features
47 Summary of Desirable Metacomputing Features
48 Status of CMS Packages - Basic Problems
49 Some Current Metacomputing Projects
50 Legion - Overview
51 Legion - Introduction
52 Legion - Philosophy
53 Legion - Distributed Objects
54 The Legion Testbed
55 The Legion Testbed
56 The Legion Testbed
57 Legion Tools
58 Legion Tools
59 Legion Tools
60 Legion Tools
61 Legion Tools
62 Legion Tools
63 Legion - Applications
64 Metacomputing - Related Projects
65 Metacomputing - Related Projects
66 Metacomputing - Related Projects
67 Metacomputing - Related Projects
68 Metacomputing - Related Projects
69 Metacomputing - Related Projects
70 Metacomputing - Related Projects
71 Metacomputing - Related Projects
72 Metacomputing - Related Projects
73 Metacomputing - Related Projects
74 Metacomputing - Related Projects
75 WANE - Wide Area Networked Environment
76 WANE - Wide Area Networked Environment
77 WANE - Wide Area Networked Environment
78 WANE - Wide Area Networked Environment
79 WANE - Wide Area Networked Environment
80 WANE - Wide Area Networked Environment
81 WANE - Wide Area Networked Environment
82 WANE - Wide Area Networked Environment
83 WANE - Wide Area Networked Environment
84 WANE - Wide Area Networked Environment
85 Near and Future Projects
86 Near and Future Projects
87 Near and Future Projects
88 Near and Future Projects
89 Near and Future Projects
90 Near and Future Projects
91 Near and Future Projects
92 Near and Future Projects
93 Near and Future Projects
94 Near and Future Projects
95 Near and Future Projects
96 Near and Future Projects
97 Near and Future Projects
98 Near and Future Projects
99 Near and Future Projects
100 Near and Future Projects
101 Near and Future Projects - MetaWeb
102 Near and Future Projects - MetaWeb
103 Near and Future Projects - MetaWeb
104 Near and Future Projects - MetaWeb
105 Near and Future Projects - MetaWeb
106 Near and Future Projects - MetaWeb
107 Near and Future Projects - MetaWeb
108 Metacomputing in the Future
109 Metacomputing in the Future
110 Metacomputing in the Future
111 Metacomputing in the Future

Outside Index Summary of Material

HTML version of Basic Foils prepared August 4 1996

Foil 1 Metacomputing: The Informal Supercomputer

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Mark Baker
Northeast Parallel Architectures Center
Syracuse University
111 College Place
Syracuse, NY 13244-4100, USA
tel: +1 (315) 443 2083
fax: +1 (315) 443 1973
email: mab@npac.syr.edu
URL: http://www.npac.syr.edu/
HTML version of Basic Foils prepared August 4 1996

Foil 2 Overview

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Parallel and Distributed Computing
Distributed Systems: Some Problems
The Challenge
Understanding the Functionality of a Metacomputer
Workings of Typical Cluster Management Software
HTML version of Basic Foils prepared August 4 1996

Foil 3 Overview

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Features of Metacomputing Management Software
  • - Computer Environments Supported- Application Support- Job Scheduling and Allocation Policy- Configurability- Dynamics of Resources
Status of CMS Packages - Basic Problems
HTML version of Basic Foils prepared August 4 1996

Foil 4 Overview

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Some Current Metacomputing Projects
  • - Legion (UVa)- Wide Area Metacomputer Manager (Italy)- Wide Area Network Environment (FSU)
Near and Future Projects
  • - WWW/CGI - RSA Factoring- JAVA based systems - MetaWeb.
Metacomputing in the future !
HTML version of Basic Foils prepared August 4 1996

Foil 5 Alternative Supercomputing Resources

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Vast numbers of under utilised workstations available to use.
Huge numbers of unused processor cycles and resources that could be put to good use in a wide variety of applications areas.
Reluctance to buy Supercomputer due to their cost and short life span.
Distributed compute resources fit better into todays funding model.
HTML version of Basic Foils prepared August 4 1996

Foil 6 Parallel/Distributed Computing - Communications Characteristics

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Parallel Computing
Communication - high bandwidth and low latency.
Low flexibility in messages (point-to-point).
Distributed Computing
Communication can be high or low bandwidth.
Latency typically high -- can be very flexible messages involving fault tolerance, sophisticated routing, etc.
HTML version of Basic Foils prepared August 4 1996

Foil 7 Some Comments about Parallel and Distributed Computing

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Why use Distributed Computing Techniques ?
Expense of buying, maintaining and using traditional MPP systems.
Rapid increase in commodity processor performance.
Commodity networking technology (ATM/FCS/SCI) of greater than 200 Mbps at present with expected Gbps performance in the very near future.
The pervasive nature of workstations in academia and industry.
Price/Performance of using existing hardware/software.
HTML version of Basic Foils prepared August 4 1996

Foil 8 Communications Performance of Some Parallel and Distributed Systems

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Comms1 - From ParkBench Suite
HTML version of Basic Foils prepared August 4 1996

Foil 9 Distributed Systems: Some Problems

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
High Initial and Maintenance Costs
  • - Cost of distributed software - Systems support staff - Technical expertise
Applications Development
  • - Relatively immature technology with few standards- Immature software development tools - Applications difficult and time consuming to develop - Difficult to tune and optimise across all platforms- Load balancing
HTML version of Basic Foils prepared August 4 1996

Foil 10 Distributed Computing Systems

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Two Types of Environment
1) Management Software
- Configured outside the OS Kernel.- Interacts with the OS.- Used to manage user applications.
2) Distributed Programming Environment
- Configured within kernel or as a dynamic kernel-module.- low-level interface necessary to get performance.- Typically used to support environments such as VSM.
HTML version of Basic Foils prepared August 4 1996

Foil 11 Management Software for Distributed Systems

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
There is not a universally used or mandated environment with which to implement, manage and run a Metacomputer at this time.
Presently there are only relatively small integrated local systems (LAN-based) - customised for local circumstances and conditions.
To create the infrastructure for a Metacomputer you would ideally like to call-up your local computer vendor and buy a package that does everything that you want!
HTML version of Basic Foils prepared August 4 1996

Foil 12 Management Software for Distributed Systems: The Problem

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
The reality is that you will need to:
- Buy/acquire the hardware.- Buy/acquire/develop or create the necessary software.- Integrate the hardware and software into a coherent system.- Edit/debug/tune/optimise your application before you can run it.- Support, maintain and "port" as time goes by.
HTML version of Basic Foils prepared August 4 1996

Foil 13 Distributed Programming Paradigms

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Support for all Programming Paradigm...
- Sequential - monolithic.- Sequential - with forked processes.- Parallel - Shared objects (Legion).- Parallel - Virtual Shared Memory (TreadMarks).- Parallel - Shared task space (Linda).- Parallel - Shared memory using directives (HPF).- Parallel - Message passing interface (MPI).- Parallel - Message passing Environment (PVM).- Parallel - Thread based/object parallelism (Java).
HTML version of Basic Foils prepared August 4 1996

Foil 14 The Challenge

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Transparent Utilisation of a Distributed
Heterogeneous Computing Environment
Want to fully utilise a heterogeneous computing environment where different types of processing resources and inter-connection technologies are effectively and efficiently used.
Fully Utilise Available Resources
Low utilisation rates of high-performance workstations (LLNL/Los Alamos 7- 10%), as their performance grows utilisation will become worse.
HTML version of Basic Foils prepared August 4 1996

Foil 15 The Challenge

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Build a Metacomputer
The use of distributed resources in this framework is known as Metacomputing and such an environment has the potential to maximise performance and cost effectiveness of a wide range of scientific and distributed applications.
HTML version of Basic Foils prepared August 4 1996

Foil 16 Understanding the Functionality of a Metacomputer

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Do Not Want to Reinvent the "Wheel", So Must...
Understand what we are trying to achieve - through-put and/or processor utilisation !?
Learn from experiences with current LAN-based Cluster Management Software (CMS) packages
Extend existing knowledge to design and develop a WAN-based Metacomputing Management package.
Use new and emerging technologies to help solve some of the existing problems.
HTML version of Basic Foils prepared August 4 1996

Foil 17 Existing Cluster Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Becoming a more common means of increasing the throughput of user applications in the US and Europe.
A significant number of packages exist - almost all originate from research projects, many have now been taken-up/adopted by commercial vendors.
Importance can be seen by both the commercial take-up and also by the widespread installation of this software at most of the major computing facilities around the world.
HTML version of Basic Foils prepared August 4 1996

Foil 18 Existing Cluster Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Much effort is being expended to increase throughput by load balancing the work that needs to be done.
Most packages are designed to run with Unix. Some support Linux (PCs) - NT support is planned by many vendors.
WWW software and HTTP protocols could clearly be used as part of an integrated management package.
Little software of this type so far been developed - several packages use a WWW browser as an alternative GUI.
HTML version of Basic Foils prepared August 4 1996

Foil 19 Some Currently Available CMS Packages

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
http://www.npac.syr.edu/techreports/hypertext/sccs-748/index.html
Commercial Packages
HTML version of Basic Foils prepared August 4 1996

Foil 20 Some Currently Available CMS Packages

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Research Packages
HTML version of Basic Foils prepared August 4 1996

Foil 21 The Workings of Typical Cluster Management Software - 1

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Step 1 - Job Description File
Produce some type of resource description file.
ASCII text file (produced using a normal text editor or with the aid of a GUI) which contains a set of keywords to be interpreted by the CMS.
The nature and number of keywords available depends on the CMS package, but will at least include the job name, the maximum runtime and the desired platform.
HTML version of Basic Foils prepared August 4 1996

Foil 22 The Workings of Typical Cluster Management Software - 2

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Step 2 - Submit Job
Job description file is sent by the client software resident on the user's workstation, to a master scheduler.
The Master Schedular
HTML version of Basic Foils prepared August 4 1996

Foil 23 The Workings of Typical Cluster Management Software - 3

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
The Master Schedular
On each of the resource workstation daemons are present that communicate their state at regular intervals to the master scheduler.
One of the tasks of the master scheduler is to load balance the the resources that it is managing.
When a job is submitted it not only has to match the requested resources with those that are available, but also needs to ensure that the resources being used are load balanced.
HTML version of Basic Foils prepared August 4 1996

Foil 24 The Workings of Typical Cluster Management Software - 4

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Multiple Queues
Typically multiple queues, each being appropriate for different types of job - for example:
  • - Homogeneous, used to service parallel jobs, - Powerful server for CPU intensive jobs, - Jobs that need a rapid turnaround.
The number of possible queue configurations will depend on the profile of the typical throughput of jobs on the system being managed.
HTML version of Basic Foils prepared August 4 1996

Foil 25 The Workings of Typical Cluster Management Software - 5

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Fault Tolerance
The master scheduler is also tasked with the responsibility of ensuring that jobs complete successfully.
It does this by monitoring jobs until they successfully finish.
If a job fails, due to problems other than an application runtime error, it will reschedule the job to run again.
HTML version of Basic Foils prepared August 4 1996

Foil 26 The Workings of Typical Cluster Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
HTML version of Basic Foils prepared August 4 1996

Foil 27 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Computing Environments Supported
Platforms Supported
  • - Heterogeneous platform support.
Operating Systems
  • - Heterogeneous OS support.
Additional Hardware/Software
  • - Is there a need for additional hardware or software !? - DCE/AFS/NIS/NFS/etc.
HTML version of Basic Foils prepared August 4 1996

Foil 28 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Application support
Batch/Interactive Job Support
- For example, a debugging session or a job that requiresuser command-line input.
Application Programming Support
HTML version of Basic Foils prepared August 4 1996

Foil 29 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Application support
Queue Type
Suport for multiple, configurable, queues.- This feature is necessary for managing large multi-vendor clusters where jobs ranging from short interactivesessions to compute intensive parallel applications needto run.
HTML version of Basic Foils prepared August 4 1996

Foil 30 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Job Scheduling and Allocation Policy
Dispatching Policy
HTML version of Basic Foils prepared August 4 1996

Foil 31 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Job Scheduling and Allocation Policy
Minimise Impact on a Workstation and Owner
- suspend jobs, change priority (nice), migrate jobs, etc.- Undesirable impact when a job is suspended, checkpointed or migrated.- Process migration requires a job to saves its state and isthen physically moved over the network to another w/s.- Impact on CPU/memory/diskspace while state is savedand on network when Mbytes of data is transferred.
HTML version of Basic Foils prepared August 4 1996

Foil 32 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Job Scheduling and Allocation Policy
Load Balancing
Load balances the resources that it is managing. - Customise the default configuration to suit the localconditions - in the light of experience...
HTML version of Basic Foils prepared August 4 1996

Foil 33 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Job Scheduling and Allocation Policy
Check Pointing
  • Save state at regular intervals during job execution. If theworkstation fails then the job can be restarted at its lastcheckpointed position.
Useful, but can be costly in terms of resources- Small job - rerun- Time critical job - necessary- Large jobs - costly
HTML version of Basic Foils prepared August 4 1996

Foil 34 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Job Scheduling and Allocation Policy
Checkpointing Needs
Additional diskspace per workstation
Filestore may be remotely mounted, impact on NFS performance and the network bandwidth.
Existing systems may not have the resources (local diskspace).
HTML version of Basic Foils prepared August 4 1996

Foil 35 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Job Scheduling and Allocation Policy
Process Migration
  • Migrating executing processes from one w/s to another.
- Minimise impact on w/s - owner takes back control.- Suspend job and then migrate it onto another w/safter a certain time interval. - Load balancing -- heavily -> lightly loaded systems.- Complicated on anything other for sequential jobs.- Similar impact to checkpointing + large state files movedaround the network - impact on network.
HTML version of Basic Foils prepared August 4 1996

Foil 36 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Job Scheduling and Allocation Policy
Job Monitoring and Rescheduling
- Monitor that jobs are running and in the event of a jobfailure should reschedule job.
Suspension/Resumption of Jobs
- Help minimise the impact of a jobs on w/s owner. - Useful in the event of a system or network wide problem.
HTML version of Basic Foils prepared August 4 1996

Foil 37 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Configurability
Resource Administration
  • Control over the resources available.
- Administrator should control who has access to whatresources and also what resources are used (CPU load,diskspace, memory).
Job Runtime Limits
  • - Enforce job runtime limits.
HTML version of Basic Foils prepared August 4 1996

Foil 38 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Configurability
Process Management
Configure the resources and manage jobs
- Control over the number of processes running.- Exclusive access to a resource by a particular job.- Control the priority of jobs.- Management and control of forked child processes.
HTML version of Basic Foils prepared August 4 1996

Foil 39 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Configurability
Job Scheduling Control
- User/administrator can schedule when a job will be run.
GUI/Command-line
- Dramatic increase in usage and popularity of the HTTPprotocol and the WWW, so a GUI based on this technologyseems likely to be a common standard in the future.
HTML version of Basic Foils prepared August 4 1996

Foil 40 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Configurability
Ease of Use
  • Easy/intuitive to work with by user/administrators.
User Allocation of Jobs
  • - User specifies the resources that they require, forexample, the machine type, job length and diskspace.
User Job Status Query and Statistics
HTML version of Basic Foils prepared August 4 1996

Foil 41 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Configurability
Scalability
The system should be scalable:- Across administrative boundaries.- Reduce SPF and increase resilience.- Practically based on "domains" but scalable fromthousands to tens of thousands of machines.
HTML version of Basic Foils prepared August 4 1996

Foil 42 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Dynamics of Resources
Runtime Configuration
Reconfigure dynamically at runtime - resources available,queues and other configurable features i.e. not necessaryrestart
Dynamic Resource Pool
- Add and withdraw resources dynamically during runtime.
HTML version of Basic Foils prepared August 4 1996

Foil 43 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Dynamics of Resources
Single Point of Failure (SPF)
HTML version of Basic Foils prepared August 4 1996

Foil 44 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Dynamics of Resources
Fault Tolerance
- System should check that resources are available beforesubmitting jobs.- Rerun a job after a workstation has crashed.- Guarantee that a job will complete. - Automatically recover status and continue to run afterfailure.- Level of fault tolerance is determined by the servicebeing provided.
HTML version of Basic Foils prepared August 4 1996

Foil 45 Desirable Features of Metacomputing Management Software

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Dynamics of Resources
Security Issues
- Provide at least normal Unix security.- Takes advantage of NIS and other industry standardpackages.
HTML version of Basic Foils prepared August 4 1996

Foil 46 Summary of Desirable Metacomputing Features

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Truly heterogeneous platforms support, across:
  • - OS and platforms
Good documentation (on-line and manual)
Vendor support
Plug and Play Installation
Batch and interactive usage
Easy to use GUI
Scalable
Easy maintenance and support
HTML version of Basic Foils prepared August 4 1996

Foil 47 Summary of Desirable Metacomputing Features

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Easy to manage, reconfigure and administrate
Ease of submitting, monitoring and control jobs
Support for all programming paradigm
Security
Statistics
No Single point of Failure
Fault tolerant
Checkpointing/Process-Migration
HTML version of Basic Foils prepared August 4 1996

Foil 48 Status of CMS Packages - Basic Problems

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
LAN-Based - not scalable!
Limited platform and operating system support - not truly heterogeneous
Do not support all programming paradigms
Load Balancing is generally naive
Single-points-of-failure
Limited-fault tolerance
HTML version of Basic Foils prepared August 4 1996

Foil 49 Some Current Metacomputing Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Projects
Legion - University of Virginia
WAMM - Italian Research Labs
WANE - Florida State University
HTML version of Basic Foils prepared August 4 1996

Foil 50 Legion - Overview

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
The Legion project is an attempt to design and build system services that provide the illusion of a single virtual machine.
Legion targets wide-area assemblies of workstations and supercomputers.
Developed at the University of Virginia.
HTML version of Basic Foils prepared August 4 1996

Foil 51 Legion - Introduction

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
It aims to provide -
- Shared-object and shared-name spaces, - Application adjustable fault-tolerance, - Improved response time and greater throughput, - Wide-area network support, - Management and exploitation of heterogeneity, - Security, - Scheduling, - Resource management, - Parallel processing- Object inter-operability.
HTML version of Basic Foils prepared August 4 1996

Foil 52 Legion - Philosophy

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Legion is an object-oriented system - designed around C++
The principles of the object-oriented paradigm are the foundation for the construction of Legion; the following features are exploited:
- Encapsulation- Inheritance- Software reuse, - Fault containment - classes- Reduction in complexity.
HTML version of Basic Foils prepared August 4 1996

Foil 53 Legion - Distributed Objects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
HTML version of Basic Foils prepared August 4 1996

Foil 54 The Legion Testbed

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Campus Wide Virtual Computer (CWVC)
The CWVC is a heterogeneous distributed computing environment built on top of Mentat.
The CWVC is used to demonstrate the benefits of a Legion-like system - provides departments at UV with an interface to high performance distributed computing.
The CWVC allows researchers at UV to share resources and to develop applications that will be usable in a true Legion setting.
HTML version of Basic Foils prepared August 4 1996

Foil 55 The Legion Testbed

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Organisations
The CWVC is currently used by organisations at UV and the NASA Langley Research Center.
Hardware Resource
CWVC contains more than 100 workstations of varying types, including - IBM/Sun/HP/SGI.
Mixture of network technologies, including ATM, FDDI, and Ethernet.
HTML version of Basic Foils prepared August 4 1996

Foil 56 The Legion Testbed

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Legion Tools
The CWVC provides a set of tools that facilitate application
development, debugging, and resource management.
Mentat
Federated File System
Thermostat
Resource Accounting Service
MAD
Prophet
HTML version of Basic Foils prepared August 4 1996

Foil 57 Legion Tools

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Mentat
Parallel C++ : The Mentat object-oriented programming language.
Provides a high-level abstractions that masks the complex aspects of parallel programming, including communication, synchronisation, and scheduling, from the programmer.
Allows the programmer to concentrate on their application.
HTML version of Basic Foils prepared August 4 1996

Foil 58 Legion Tools

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Federated File System (FFS)
Objects that execute on hosts in the CWVC are presented with a single unified file system abstraction - FFS.
The FFS interacts with local file systems, i.e. NFS mount structures, etc., so files visible on one host may not be available on another.
The FFS allows objects to view a single, unified file name space, and thus execute in a location independent manner.
Interfaces is similar to the Unix standard library file system - little change to existing code necessary.
HTML version of Basic Foils prepared August 4 1996

Foil 59 Legion Tools

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Thermostat
Thermostat provides an GUI to manage resources.
Allows a resource owner to schedule the times of day and the days of the week that hosts will be available for CWVC use.
It also allows the resource owner to specify the percent of the CPU time and memory that may be used by the CWVC during individual available time slots.
HTML version of Basic Foils prepared August 4 1996

Foil 60 Legion Tools

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Resource Accounting Services
Resource utilisation is logged on a per-user basis, while resource availability (controlled via the Thermostat) is logged on a per machine basis.
Usage and machine availability "credits" are scaled based on the computing power of the hosts involved - time on an SP-2 node is worth more time than on a Sun IPC. A report generation tool is provided to extract and summarise usage statistics.
HTML version of Basic Foils prepared August 4 1996

Foil 61 Legion Tools

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
MAD
MAD is a set of tools that enables programmers to debug their CVWC applications with the debugger of their choice.
MAD supports post-mortem debugging: CWVC programs run to completion, or until an error
The programmer can replay a specific object, i.e. reproduce its execution, under the control of a sequential debugger, and can use the traditional debugging cycle to find the bug.
HTML version of Basic Foils prepared August 4 1996

Foil 62 Legion Tools

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Data Parallel Computation Scheduler - Prophet
Automatic run-time scheduling system for SPMD computations.
Helps with processor selection, data decomposition, task placement.
Chooses the best subset of processors to use based on the problem computation granularity.
Decomposes data to provide processor load balance.
Assigns tasks to processors to limit communication overhead.
HTML version of Basic Foils prepared August 4 1996

Foil 63 Legion - Applications

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Scientific Applications
Genome Library Comparison
Atmospheric Simulation
Automatic Test Pattern Generation for Integrated Circuits
Electrical Engineering has developed a parallel Automatic Test Pattern Generation (ATPG) application.
URL http://www.cs.virginia.edu/~legion
HTML version of Basic Foils prepared August 4 1996

Foil 64 Metacomputing - Related Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WAMM (Wide Area Metacomputer Manager)
WAMM is a graphical tool, built on top of PVM.
Provides user with a GUI to assist in tasks such as: host add, check, removal, process management, compilation on remote hosts, remote commands execution.
HTML version of Basic Foils prepared August 4 1996

Foil 65 Metacomputing - Related Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WAMM (Wide Area Metacomputer Manager)
Sites Involved (Italy)
CINCECA - Interuniversity Consortium of Northeast Italy for Automatic Comp - Bologna
CASPUR - University and Research Consortium for Supercomputing Apps - Rome
CRS4 - Centre for Advanced Studies, Research and Development - Sardinia
CNUCE - institute of the Italian National Research Council - Pisa
ScuolaNormale Superiore - Pisa
Connection
Networked by GARR, the Italian research network - 2 Mbps.
HTML version of Basic Foils prepared August 4 1996

Foil 66 Metacomputing - Related Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WAMM (Wide Area Metacomputer Manager)
HTML version of Basic Foils prepared August 4 1996

Foil 67 Metacomputing - Related Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WAMM (Wide Area Metacomputer Manager)
GUI
All functions are accessible via menus and buttons.
Geographical view of the system.
Hosts are grouped following a tree structure - WAN, MAN and LAN).
HTML version of Basic Foils prepared August 4 1996

Foil 68 Metacomputing - Related Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WAMM (Wide Area Metacomputer Manager) - WAMM Tree
HTML version of Basic Foils prepared August 4 1996

Foil 69 Metacomputing - Related Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WAMM (Wide Area Metacomputer Manager)
Remote Command Execution
UNIX commands (e.g. ls, uptime, who, etc.) as well as X11 programs (e.g. xload, xterm, etc.) can be executed on remote hosts.
WAMM takes care of showing command output (for UNIX ones) and windows (for X11 ones) on the user's display.
HTML version of Basic Foils prepared August 4 1996

Foil 70 Metacomputing - Related Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WAMM (Wide Area Metacomputer Manager)
Remote Compilation
Compilation of modules on remote nodes is greatly simplified.
The user selects a group of hosts to compile onto and a set of source files to be compiled.
WAMM copies sources on remote nodes, compiles them in parallel and shows progress in separate windows, one for each host.
HTML version of Basic Foils prepared August 4 1996

Foil 71 Metacomputing - Related Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WAMM (Wide Area Metacomputer Manager)
Configuration
The Metacomputer configuration is specified through an external file, written in a simple declarative language.
Number and grouping of hosts, remote commands for each node, icons can be specified.
Graphical aspect (colours, fonts, etc.) can be customised via standard X11 resource files.
HTML version of Basic Foils prepared August 4 1996

Foil 72 Metacomputing - Related Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WAMM (Wide Area Metacomputer Manager)
HTML version of Basic Foils prepared August 4 1996

Foil 73 Metacomputing - Related Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WAMM (Wide Area Metacomputer Manager)
Software Requirements
PVM version 3.3 or higher
X11 Release 5 or higher
Motif version 1.2 or higher
XPM version 3.4 or higher
HTML version of Basic Foils prepared August 4 1996

Foil 74 Metacomputing - Related Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WAMM (Wide Area Metacomputer Manager)
Supported Platforms
HP/Sun/IBM/SGI/Digital.
URL http://miles.cnuce.cnr.it/pp/wamm/
HTML version of Basic Foils prepared August 4 1996

Foil 75 WANE - Wide Area Networked Environment

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Introduction
WANE is a Wide Area Networked Environment being developed at the Supercomputer Computations Research Institute (SCRI) as part of a three year grant from the US Department of Energy.
It is designed to be provide highly scalable, fault tolerant computing and information encompassment.
WANE is the integration of several ongoing software projects into one single transparent environment.
HTML version of Basic Foils prepared August 4 1996

Foil 76 WANE - Wide Area Networked Environment

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Key technologies exploited by WANE
- DQS - Job scheduling - PostGres - database accesses - Tcl/Tkl - X-Window development - Mosaic/WWW/gopher - internet information services - FreeNet - information services
All software developed in this project will be supported as a "best-effort" by SCRI.
HTML version of Basic Foils prepared August 4 1996

Foil 77 WANE - Wide Area Networked Environment

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WANE is a comprehensive Internet service package providing all the underlying software necessary to connect a server to the Internet.
Furthermore, WANE provides the client software for a variety of platforms, allowing users to access all Internet services.
WANE is a way for users to connect to the Internet quickly and without great expense or problems.
HTML version of Basic Foils prepared August 4 1996

Foil 78 WANE - Wide Area Networked Environment

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
The Wane Server
The server software package is a turn key solution targeted at a diverse audience, which includes commercial, governmental, educational, and individual users.
The system is designed to support thousands of users, is scalable, and fault tolerant.
The package is modular, with services ranging from a simple mail or WWW server to a full fledged free net service package.
HTML version of Basic Foils prepared August 4 1996

Foil 79 WANE - Wide Area Networked Environment

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
The server package contains:
Extensive user access controls.
Hierarchical administrative levels.
Multiple user domains.
Optional support of separate administrative groups, allowing delegation of administrative tasks.
User friendly GUI.
Administration.
A complete distribution of Linux.
HTML version of Basic Foils prepared August 4 1996

Foil 80 WANE - Wide Area Networked Environment

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Multi-user operating system on inexpensive PC clones.
Mail Hub service
World Wide Web multimedia server software
Internet connection tools/SW
Internet Newsgroups
Comprehensive suite of interactive personal
Communication tools (Text, Audio, Video)
Access to vast archives of software and data
HTML version of Basic Foils prepared August 4 1996

Foil 81 WANE - Wide Area Networked Environment

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Client Software:
The client software includes a variety of public domain and shareware packages enabling users to connect to the Internet via modem or LAN.
The WANE client distribution provides support for a diverse set of platforms, including Macintosh, DOS, Windows, Windows NT, OS/2, and Unix.
The connectivity tools provided allow full multimedia access over both LAN and phone lines.
HTML version of Basic Foils prepared August 4 1996

Foil 82 WANE - Wide Area Networked Environment

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Client Software
The Client distribution includes:
- Network connection packages - PPP and SLIP.
- WWW browsers.
- Electronic mailer programs.
Internet News readers
HTML editor - Hot Metal, XHTML Edit & Phoenix
Optional menu based environments (graphical and text) for administration and/or novice users
HTML version of Basic Foils prepared August 4 1996

Foil 83 WANE - Wide Area Networked Environment

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Typical WANE Server Usage Diagram
HTML version of Basic Foils prepared August 4 1996

Foil 84 WANE - Wide Area Networked Environment

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
URL - http://www-wane-leon.scri.fsu.edu/
HTML version of Basic Foils prepared August 4 1996

Foil 85 Near and Future Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WWW/CGI Computing
WWW is now the most promising candidate for the universal access core component of the NII.
Current Web is ~15,000 servers and expands at the rate of ~1 new server/hour.
Software industry starts adding value (Netscape, Netsite, Mosaic licenses, HotMetal, Netforce, Web support in OS/2 Warp and Windows95)
.
HTML version of Basic Foils prepared August 4 1996

Foil 86 Near and Future Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WWW/CGI Computing
So far, Web was mainly used for static hypermedia such as local information pages, digital libraries, Internet directories etc. However, the WWW model offers also extension mechanisms (CGI, CCI) towards dynamic services and in fact arbitrary computation.
Early interactive Web services appearing. Examples include: WebCalc (NASA Goddard), Easy HTML (NCSA), WebChat (Internet Society), Virtual Doors (Unique, Inc.), Visioneering's Imaging Machine (VRL, Inc.).
HTML version of Basic Foils prepared August 4 1996

Foil 87 Near and Future Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WWW/CGI Computing
Key points in Web Technology: Characteristics
Current main components: HTTP; HTML; CGI; Fillout Form
Client-server communication model - (Flat hierarchical UNIX) File system as the major file (data) management system
HTML version of Basic Foils prepared August 4 1996

Foil 88 Near and Future Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WWW/CGI Computing
Key points in Web Technology: Strengths
Established Internet as the major vehicle in networking industry
Universal, hyperlinked information access and dissemination
Transparent networking navigation and GUI with multimedia information access for information - dissemination.
HTML version of Basic Foils prepared August 4 1996

Foil 89 Near and Future Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WWW/CGI Computing
Key points in Web Technology: Weaknesses
Static, browser-oriented client.
Document update done manually, hard to automate.
Flat UNIX file system supports only primitive information system functions such as open, read/write and close.
HTML version of Basic Foils prepared August 4 1996

Foil 90 Near and Future Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WWW-Based Project Undertaken at NPAC
Collaboration with Boston University and Cooperating Systems NPAC has been developing concepts and prototypes of Compute-Webs.
Partly motivated by the integration of information processing and computation for both a better programming environment and for a natural support of data intensive computing.
Web represents the largest available computer with worldwide some 20 million potential nodes which is expected to grow by an order of magnetude as the Superhighway is deployed fully.
HTML version of Basic Foils prepared August 4 1996

Foil 91 Near and Future Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WWW-Based Project Undertaken at NPAC
First prototype was built on compute-extended Web Servers using the standard CGI mechanism and applied successfully to the factorisation of the RSA 130 digit number which was distributed to a net of Web servers.
This work was presented at the SC'95 and was given the award as the most geographically dispersed and heterogeneous metacomputing solution in the Teraflop Challenge contest.
HTML version of Basic Foils prepared August 4 1996

Foil 92 Near and Future Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WWW-Based Project Undertaken at NPAC
RSA Factoring Challenge
Public-key cryptosystem for both encryption and authentication; invented in 1977 by Rivest-Shamir-Adleman (RSA).
RSA a cryptosystem where each party has two keys: a public key and a corresponding secret key.
The public key is made public, the secret key is kept secret.
HTML version of Basic Foils prepared August 4 1996

Foil 93 Near and Future Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WWW-Based Project Undertaken at NPAC
RSA Factoring Challenge
In RSA the secret key can be derived from the public key.
Factoring large numbers is believed to be hard - so RSA is believed to be secure.
HTML version of Basic Foils prepared August 4 1996

Foil 94 Near and Future Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WWW-Based Project Undertaken at NPAC
RSA Factoring Challenge
People are protecting their data and money using 155- digit (i.e., 512-bit) numbers.
WWW-factoring project is the first large scale project that makes use of a new and factoring method - the Number Field Sieve (NFS).
First goal was to factor RSA 130 - ultimate goal to to break a RSA 155.
HTML version of Basic Foils prepared August 4 1996

Foil 95 Near and Future Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WWW-Based Project Undertaken at NPAC
RSA Factoring Components - FAFNER
FAFNER is a collection of Perl scripts, HTML pages, and associated documentation which together comprise the server-side of the Web factoring effort.
The FAFNER software provides interactive registration, task assignment, and solution database services to sieving clients.
HTML version of Basic Foils prepared August 4 1996

Foil 96 Near and Future Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WWW-Based Project Undertaken at NPAC
GNFS (General Number Field Sieve)
The GNFS client package implements the sieving algorithms that converts a task specification into a set of useful results (called relations).
GNFS performs relatively little I/O - embarrassingly parallel, and has large (but configurable and constant for an entire run) memory requirements.
HTML version of Basic Foils prepared August 4 1996

Foil 97 Near and Future Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WWW-Based Project Undertaken at NPAC
GNFSD (General Number Field Sieving Daemon)
GNFSD is an augmented sieving client that allows a GNFS process to interact with a task server over the net, rather than requiring task specification on the command line.
Other key features are automatic failure detection and restart via a watchdog timer, persistent configuration state, and a TCP/IP monitor interface at port 5453.
HTML version of Basic Foils prepared August 4 1996

Foil 98 Near and Future Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WWW-Based Project Undertaken at NPAC
How It All Fits Together
The FAFNER servers are hierarchical; root server, plus several major subservers - each in turn has subservers, and so forth.
Subserver depends on its parent for sieving tasks.
The sieving clients (GNFS or GNFSD) are the leaves of the FAFNER tree; they get a single task from a FAFNER server, and then spend time computing the problem.
HTML version of Basic Foils prepared August 4 1996

Foil 99 Near and Future Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WWW-Based Project Undertaken at NPAC
When the answers are ready (in the form of a text file containing a few 100 or few 1000 relations), the clients send them back to their FAFNER server.
There, they are distilled, archived, and ultimately sent back to Bellcore, where they are integrated into the final solution - the factoring of RSA number.
Successfully computed RSA-130.
HTML version of Basic Foils prepared August 4 1996

Foil 100 Near and Future Projects

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
WWW-Based Project Undertaken at NPAC
The Problem with Fafner
Major problem with the CGI enhanced Web servers that supported RSA factoring, was that they did not provide the standard support which one expects from clustered computing packages.
Such as - load balancing, fault tolerance, process management, automatic minimisation of job impact on user workstations, security, and accounting support.
HTML version of Basic Foils prepared August 4 1996

Foil 101 Near and Future Projects - MetaWeb

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
A Scalable Metacomputer Management Package
The overall aim of this project is to design, develop and implement a WWW-based Metacomputer management package - MetaWeb.
Project will build on existing knowledge and experiences with the management of LAN-based computing clusters to produce a software package capable of managing a potentially globally distributed Metacomputer.
HTML version of Basic Foils prepared August 4 1996

Foil 102 Near and Future Projects - MetaWeb

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Objective is to increase the through-put of user applications by utilising the wealth of existing networked computing resources efficiently and effectively together.
Ecourage collaboration between groups using MetaWeb.
Truly heterogeneous, capable of managing resources ranging from PCs to vector/MPP supercomputers.
Capability will be based on the use of pervasive WWW software and take advantage of Java architecturally neutral ByteCode.
HTML version of Basic Foils prepared August 4 1996

Foil 103 Near and Future Projects - MetaWeb

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
MetaWeb: designed to be fully fault tolerant.
Not only will it be able to reboot itself and retain its previous status but also be able to resume or restart failed application jobs.
This ability is enabled by the fully duplicated design of MetaWeb and also by the use of a persistent database to maintain the Metacomputer's current status
HTML version of Basic Foils prepared August 4 1996

Foil 104 Near and Future Projects - MetaWeb

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
The MetaWeb Prototype
Using existing WWW-based technologies such as Perl CGI-scripts, C-modules and HTTP servers.
This prototyping phase of the project will allow the MetaWeb design be proven and adapted as necessary.
Ensure that the implemented version of the package will be functional, robust and work as it is intended to.
HTML version of Basic Foils prepared August 4 1996

Foil 105 Near and Future Projects - MetaWeb

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
MetaWeb - The Product.
MetaWeb will replace the need to use existing research and commercial cluster management packages, such as Codine, LoadLeveler, DQS, etc, by exploiting emerging technologies and the ubiquitous nature of WWW.
MetaWeb will exhibit all the best features of the existing management packages but will have the advantage of being specifically designed and developed with all the latest and emerging WWW technologies at hand.
HTML version of Basic Foils prepared August 4 1996

Foil 106 Near and Future Projects - MetaWeb

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
MetaWeb Project Collaboration
Northeast Parallel Architectures Center
Cornell Theory Center
  • - David Lifka (Author of EASY)- Joe Skovira (IBM - LL)
  • Genias - GmbH (Codine)
HTML version of Basic Foils prepared August 4 1996

Foil 107 Near and Future Projects - MetaWeb

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
HTML version of Basic Foils prepared August 4 1996

Foil 108 Metacomputing in the Future

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
The Future Trends...
Long term is hard to predict - See changes over last 5 Years!!
Can see trends, however...
HTML version of Basic Foils prepared August 4 1996

Foil 109 Metacomputing in the Future

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Hardware Trends (5 - 10 Years)
Computers
Millions (100 - 300) of "settop" boxes
One in every US household
More worldwide
Ranging from Supercomputer to Personal Digital Assistants
HTML version of Basic Foils prepared August 4 1996

Foil 110 Metacomputing in the Future

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Hardware Trends (5 - 10 Years)
Networks
Networks (1 - 20 MBytes/s) - fulfil needs of "home" entertainment industry.
Technologies ranging from high-bandwidth fibre to Electro-magnetic types such as Microwave.
HTML version of Basic Foils prepared August 4 1996

Foil 111 Metacomputing in the Future

From MetaComputing -- the Informal Supercomputer Tutorial for CRPC Annual Meeting at Argonne -- May 13 1996. *
Full HTML Index
Hardware Trends (5 - 10 Years)
Software
Very Hard to Predict in relatively short term - JAVA has been
product for about a year !!
Ubiquitous and pervasive (WWW/JAVA-like).
Can forget about underlying h/w and OS.
Metacomputing "plug-ins"
Micro-kernel-like JAVA based servers with add-on services that can support Metacomputing (load balancing, migration, checkpointing, etc...).
Northeast Parallel Architectures Center, Syracuse University, npac@npac.syr.edu

If you have any comments about this server, send e-mail to webmaster@npac.syr.edu.

Page produced by wwwfoil on Sun Apr 11 1999