A National Software Exchange for the
High Performance Computing and Communications Program
Jack Dongarra
Geoffrey Fox
Ken Kennedy
Jim Pool
Rick Stevens
Center for Research on Parallel Computation
We propose to facilitate the development and distribution of software
enabling technologies for high performance computing by developing a
National Software Exchange (NSE) for High Performance Computing and
Communications. Key components of the project are: distribution via the
national information infrastructure, a multilevel software classification
system, a network-based catalog that will serve as a "road map" to
important HPCC enabling technologies, a system for selective promoting of
important emerging technologies, a research and development program in
advanced network-based software distribution mechanisms, an outreach and
technology transition effort and a methodology for measurement of the
success of the project.
The Center for Research on Parallel Computation will implement this system
over a five-year period, emphasizing early deployment of the simplest
mechanisms early in the cycle.
Audience
The potential audience for the NSE has three important components:
- The HPCC application and computer science community. These groups
will be a major source of material for all aspects of the NSE but they will
not be the major users. The HPCC applications community often needs to
develop highly optimized components from scratch and so generic reusable
components as in the NSE may not be so helpful. Furthermore, the Grand
Challenge program has already linked application scientists strongly with
the computer science community.
- Users of NASA, NSF, DoE and other federal centers. These users are
an important audience for the NSE because they are less knowledgeable and
less performance oriented than the first class. They will be users of good
if not optimal libraries and templates. Further we have a natural support
organization in place through the consultants at the supercomputer centers
which they use. We intend that the NSE work closely with these consultants
and other supercomputer center staff to both design improve and disseminate
the NSE.
- Other users of high performance computers. This group includes the
many current and potential industrial users for whom the NSE will be a very
valuable resource. This category presents special challenges as there is no
natural support organization to help these users.
Software Distribution via the National Information Infrastructure
The key to a successful software distribution system will be the
establishment of a scalable mechanism for distributing software via the
national information infrastructure. Many Internet sites have sizable
collections of documents or software. It is both unnecessary and
undesirable to require that these collections reside at a single site. We
plan to put together a CRPC HPCC Mosaic Home Page that will capture in one
place the software from the HPCC effort.
We plan to develop a software exchange capability for the HPCC community
based on existing network interfaces such as Mosaic. This capability is
intended to provide mechanisms for the community to access a number of
existing software.
The maximal benefit to the community will be achieved by having an
architecture for software exchange that is open. This will allow a network
repository to grow over time. Initially, we will provide existing software
technologies to the HPCC community. This will be done by building on
existing components to build a prototype that will serve as a model of how
such a system should be put together. This prototype can be in place and
operational within a month and will evolve through several stages of
experiment, prototype, and operating capability.
This prototype for the HPCC National Software Exchange will demonstrate
how software can be effectively exchanged and reused. It will be based on
the underlying premise (reflecting today's software practices) that
successful software modules and software, although they may take advantage
of pre-existing components, are written primarily by a single team.
Software is re-used either by incorporation as subroutines or by use as a
starting point for writing new code.
The system will have the following attributes:
- Speed: Retrieving software should take seconds, not minutes as
typically required by e-mail.
- Usability: The user interface should make searching through a
large collection of software and documents easy.
- Organization: The system's repository should be a moderated
collection, with up-to-date indexes, and a database organized to facilitate
searching and ease of retrieval. The repository may be distributed over
several sites.
- Record keeping: The system should have the capability of logging
requests so updates and corrections can be reported to users.
- Security: The system should be secure from accidental or
intentional misuse.
- Portability: The system's implementation should be as portable as
possible.
- Accessibility: The system should be accessible to a large number
of users.
Multilevel Software Classification System
Software in the National Software Exchange will be classified into multiple
level, representing different levels of evaluation that the software has
passed. Initially, there will be two levels:
- Level 1: software that is contributed by authors with no review
process except for a mechanical check that the software meets certain
minimum requirements such as providing simple documentation and keywords
for use in a classification system.
- Level 2: software that has been reviewed by a process to be
described shortly and certified to meet certain minimum quality and
usefulness criteria.
Eventually, higher levels of quality, and an associated review process,
can be established. Possible criteria might be robustness and level of
support provided by the authors.
The process for certification at Level 2 would be as follows: First, the
authors would propose such a move, including a proposed classification for
the software. Based on the initial classification, the review would be
assigned to one of the CRPC scientists who will serve as area editors in
the process. The area editor would then select 2 to 6 reviewers who would
review the software and provide a review report. Based on this review, the
area editor would accept the software, return it to the author for
specific improvements or reject it. Once accepted, the reviewers' reports
would be used to incorporate information about this software into the road
map described in the next section.
The following is a list of the people within CRPC who would serve as senior
area editors and would recruit other area editors as needed.
Area Editors:
- Compiler Technologies -- Chandy
- Visualization and Virtual Environments -- Messina
- Linear Algebra -- Dennis
- Numerical Simulation Technologies --
xnetlib: We propose to extend and
continue development of the netlib/xnetlib software systems and provide
complete linkage capability with Mosaic and World Wide Web. This work will
insure that the premier extant scientific and technical software repository
becomes fully integrated in the NSE.
- indexing methods: We propose to investigate new methods for
improving the indexes available through Mosaic for software packages and
libraries. In particular we will investigate the use of automatic indexing
methods to provide software interface indexes and keyword indexes that link
software descriptions and function to specific software systems. This
technology will be used to produce and maintain a comprehensive union
catalog of software and systems. (index generation mechanisms etc.)
- URN search engine and naming strategies: Naming becomes
increasingly important as the number of objects in the global repository
increases. We propose work to develop a Universal Resource Naming
mechanism that provides the basic foundation for resource indexing and
search.
- repository in a box: We propose to develop the tools to allow
anyone on the network to quickly set up a software repository for local and
global use. This tool kit will enable software developers to link in their
repository to the National HPCC Page and thereby become part of the NSE
without need of a third party.
- remote execution (f2c etc.): We propose to develop remote execution
mechanisms that will allow users to access network computing resources that
provide specialized services (e.g. Fortran to C conversion, Fortran program
transformation. etc.)
- active objects/libraries: We will develop the technology to enable
libraries once installed to automatically update themselves via the network
whenever a new version is available. This will be done such that the user
is simply notified and the interfaces are maintained. We will also provide
the capability to preserve locally modified interfaces for specific look
and feel.
- broker system based on CORBA/SOM for matching APPS and libraries
(objects): We propose to develop a network based mechanism that provides
for the automatic "arrangement" of services for users. These brokers will
match applications and libraries through existing standard naming services
link CORBA and SOM and will also provide the capability to arrange for
compute server access matching user requirements.
- Agents (HPCC version of knowbot type things.. leverage work in the
commercial area like Telescript and General Magic): Much work is currently
underway on the development of surrogate agents for network query and
search and for conducting basic tasks using the internet. We propose to
leverage the commercial developed systems and provide an "HPCC community
aware" set of agents that can perform common tasks associated with the NSE
for users. These agents will over time be able to conduct searches with
complex criteria and automatically install software and libraries.
- Mosaic-Killer-post-web-interface-thing (TBD)
- search environment development: New methods and software will be
developed to support network queries, assist the user in query formation,
system response editing and improving the search process.. This work will
be closely tied to index methods and development of search agents.
- work on technology for scalable implementations of repositories:
Work will be conducted on new methods and techniques for scaling the
software repository. These methods include replication of the repository,
dynamic load balancing of requests to "mirror" sites, automatic recovery of
site failure and index and URN/URL name space management.
- install tools (like nextstep installer): We propose to develop
software that will assist the user in installing software
packages/libraries retrieved from the NSE. These tools will allow automated
installation and verification of the installed system.
- usage collection methods and tools: Tools and software will be
developed to collect usage data and to provide mechanisms for user feedback
on the software provided by the NSE. This data will be available to the
software developers and to the management of the NSE for improving both the
process and the software systems.
Technology Transition
We described three classes of users who we expect to use the NSE. The first
class, the ongoing HPCC community, are knowledgeable enough that the NSE
will be directly usable without additional outreach efforts. The road map
included with the NSE will be sufficient for technology transition to these
users. The other users will certainly benefit from the purely electronic
NSE but we expect that additional proactive efforts will be needed for
technology transition. The second class of users are supported by existing
supercomputer centers and we propose that the NSE group proactively work
with the staff of these centers to allow them to provide consulting and
other help needed for this class of NSE user. In particular we expect that
the supercomputer centers will bring the NSE to the attention of these
users and provide us critical help in designing and implementing system to
be effective.
The remaining users in the third class do not have such a clear support
organization but we have already started two efforts that are prototypes
for supporting these users. In the first we are working with the NSF
supercomputer centers on a set of educational activities which will reach
out to the many state and regional supercomputer centers and hence to their
support staff. In this way we can extend the cadre of those qualified to
help the less sophisticated NSE user by teaching the consultants at these
regional sites.
We have also started at direct outreach to small businesses in our InfoMall
project which we can use as a prototype for other needed outreach
activities. For instance New York State has funded NPAC to provide very
proactive HPCC business consulting for the mid-Hudson region of the state.
We will use this and other initiatives to prototype outreach to
nontraditional users of supercomputer and HPCC resources.
Methodology for Measurement
We will conduct systematic surveys to determine the extent of usage of the
NSE as a whole and individual software components within the NSE. Survey
forms would be submitted electronically and would be very simple to ensure
a high return rate. They would ask questions designed to determine whether
a software component downloaded from the NSE is actually in use and, if so,
the volume of usage. We would attempt to survey all the usage of level-2
software and a randomly-chosen subset of level-1 contributions. We would
pay particular attention to usage within industry, tracking that as
accurately as possible
![xnetlib](people/fox.gif>Fox</a>
</ul>
<p>To summarize, software in Level 2 would be reviewed by a process similar to
the one used for reviewing articles submitted to a scholarly journal, with
the CRPC providing area editors for the process.
<h3>A Road Map to HPCC Enabling Technologies</h3>
<p>Based on our experience in parallel computation, CRPC will construct a
catalog or "road map" to high-performance computing enabling technologies.
This will be available via the network and will employ multimedia
techniques to support fast navigation.
<p>We intend to embed the NSE software deliverables in a technology road map
that gives context and advice to the user of HPCC reusable technology
pieces. At its simplest, the road map is a description of the various
technology components and their interfaces, with hypertext pointers to
associated software and newsgroups. For each technology we will provide an
overview authored by an expert in the field. This would be at level of a
review article in a refereed journal but designed for electronic browsing
and linkage to other NSE components. Indeed, the area editors would review
and arrange for refereeing of this material. A technology road map would
certainly include a description of area, indication as to open issues and
status of current NSE software deliverables. It will describe what type of
material is available and whether we have a full library implementation for
the component or whether the NSE includes generic templates. Further the
road map would give references (whether available in hard copy or
electronically) and possible experts and consultants that could give help.
The road map will also include multimedia educational material which could
be used to train the outreach staff discussed in the technology transition
section. We have already started the design of such material as part of our
NSF outreach project for supercomputer center support staff. Educational
material will include electronic copies of presentation material and
on-line video of experts lecturing on the material, if available.
<p>The road map can be thought of as a consumer reports for HPCC and we
expect that our refereeing procedure will ensure that any dangers
associated with bias in the road map can be minimized. We expect that the
road map will be very important in guiding HPCC development as it will
clearly delineate those areas where further work is needed and minimize
redevelopment of components that are already satisfactory. Further it will
promote good modular software design by describing the possible components
users can access and encouraging the development of standard interfaces
between components.
<h3>Promotion of Software Capitalization</h3>
<p>In many cases, the most promising technologies developed in the HPCC
community may not yet be in a form suitable for distribution at level 2 in
the National Software Exchange. This is often the case with research
prototypes developed in the academic community. We propose to augment the
NSE through a program to "capitalize" such prototype systems by providing
funds needed to bring them into a form suitable for distribution and use.
These improvements might be carried out by the originators of the
technology or, in some cases, third parties.
<p>We propose a program by which four to eight software systems would be
chosen each year for further development. Selections would be made by an
outside review committee convened by CRPC. Judgments would be based on
value of the functionality, quality of the prototype and a demonstrated
ability to finish this type of project. Each year, a substantial number of
nominations would be submitted to the review committee by CRPC. These
nominations might be made by CRPC researchers or outside investigators.
Nominations would be careful reviewed an a few selected, with the explicit
number determined by the budget. CRPC would then negotiate a subcontract
for the work.. The result would be expected to meet all the criteria for
certification in the NSE at level 2.
<h3>Research Projects for Advanced HPCC Software Exchange Technology</h3>
<p>We are proposing a set of related projects that are aimed at improving the
technology for building software/information repositories, for indexing and
search, for automating the maintenance of downloaded systems and for
improving the scalability of existing tools and methods. These research
projects will feed technology as it is developed into the NSE on an
incremental fashion, therefore maintaining the NSE as a world leading
network resource. This work will be conducted jointly by Argonne National
Laboratory and University of Tennessee as part of the CRPC managed National
HPCC Software Repository.
<ul>
<p><li>further development of netlib/<a href=){kind=link}