ARL PET Project

ARL-CY5-IC--3

Institution Name: Florida State University

Project Identifier: ARL-CY5-IC--3

Project Title: Extension of the Gateway System to Support New CTAs

POC: Fox, Geoffrey C

Email: gcf@cs.fsu.edu

Phone: 850 644 4587

Fax: 850 644 0098

CTA: IC

Project Description: Gateway a three-tier system designed to provide a seamless web-based
environment for accessing a broad range of back-end HPC services
including parallel computers, visualization systems, database systems,
etc. By providing a middleware, consisting of well-defined services
and application program interfaces (APIs), the broad range of back-end
services can be presented at a level of abstraction which separates
applications from many details of the back-end implementation. The
front-end, or user interface, connects to the middleware using its
abstractions and APIs, thus providing tremendous flexibility and
generality to the front-end. Front-ends can take many forms depending
on user needs. One example would be using the Gateway environment to
develop a complex multi-physics/multi-scaling simulation by coupling
of several HPC applications. Another example, which is the current
focus of PET Gateway activities, is the development of domain-specific
problem solving environments (PSEs) which provide a suite of tools
under a unified interface designed to assist users in a given CTA or
field with everything from input preparation to job submission and
execution, to analysis of the results.

The Gateway project, a collaboration between Florida State University
(the team having moved from Syracuse University), the Ohio
Supercomputer Center, ASC MSRC, and ARL MSRC, uses a CORBA and
XML-based middleware and modern web technologies to provide this
seamless environment. During Year 4, PET support has focused on the
development of a CCM PSE (other types of front-ends are being
developed with other funding) and on deployment of the system in
testbed environments at both ASC and ARL. These testbeds provide user
feedback and operational experience which is crucial to preparing the
system wider deployment, possibly into production environments. An
important part of our Year 5 activities will be aimed at supporting
the ARL testbed and improving the Gateway middleware to address
problems identified during testing. The other primary focus of our
Year 5 work will be the expansion include PSEs for other areas.

The CCM PSE serves as an exemplar of one common class of HPC use,
where many long-running compute-intensive jobs are executed in batch
mode, accompanied by input preparation and data analysis tasks which
are, in comparison, computationally modest in their requirements.
This is a large class of applications, which is why CCM was chosen as
the initial target, but it is by no means the only way to interact
with HPC systems. Therefore, despite conscious efforts to construct
the PSE framework and middleware interface in a general fashion, there
will inevitably be modifications required to accomodate other classes
of HPC users. Therefore, during Year 5, we plan to collaborate with
OSC and appropriate CTA teams to develop one or two additional PSEs
targeting other classes of use in order to insure the appropriate
level of generality in the PSE framework and the middleware. Such a
step is necessary before we can consider opening up the Gateway system
to application developers across the Modernization Program, inviting
them to create PSEs for their fields.

Our initial target will be the IMT CTA, which presents a very
different model of HPC use. IMT often involves large volumes of data
and requires flexibility in connecting the data with analysis tools.
It is also more interactive than batch oriented. The ARL PET team has
already developed a prototype secure web-database environment which
will serve as a starting point for a more general IMT PSE. Our plan
is to have an initial IMT PSE running on the ARL testbed and available
to users by the end of Year 5, with intermediate deliveries of
designs, prototypes, and limited pioneer user testing to be determined
as the specifics of the new PSE are fleshed out. Once work on the IMT
PSE is well underway, we may begin the development of a second new
PSE. Currently we see CSM is particularly promising because it seems
to offer a combination of feature of the CCM and IMT areas.
Substantial batch-oriented computing combined with intensive analysis
of large data sets thus produced. In particular, this field seems to
offer an excellent opportunity for the integration of data mining
tools. Work on a CSM PSE is not necessarily expected to be completed
during Year 5, but should be sufficiently advanced to serve as a
confirmation of the architectural modifications resulting from the IMT
PSE work.

This work, together with related activities during Year 5 at ASC will
provide the basis for the development of training materials to
introduce others to PSE development for the Gateway system. A
workshop on this topic is part of our I/C Training proposal, and will
help set the stage for opening up Gateway to the larger HPCMP
community.

Florida State's role in this work, as in the past Gateway work, will
focus primarily on the middleware, while OSC will focus on the PSE
framework. This project will help to refine the interface between
these two tiers of the system, and so will require the close
cooperation of the two organizations.

Project Objectives: The Gateway system, with a CCM PSE, is being deployed in testbed
environments at ASC and ARL, and will soon be open to pioneer CCM
users. The development of specific new PSEs will help to insure that
the PSE-middleware interface and middleware services are defined at
the appropriate level of abstraction and with appropriate interfaces
to accomodate the requirements of a broad range of PSEs. This is the
essential next step as we move towards the ability to deploy Gateway
as a production system at the MSRCs, and open up PSE development to
the entire HPCMP community.

Deliverables: 1) On-going support and enhancement of the ARL MSRC Gateway testbed
installation, together with ARL MSRC staff
2) Support for the development of new front-ends for the IMT and CSM
CTAs (specific CTAs subject to revision in disccusion with PET
management), in collaboration with the Ohio Supercomputer Center
and appropriate CTA team members
3) Development, in collaboration with OSC and ASC, of training
materials to help open the development of new PSEs for other CTAs

Customers/End Users: HPCMP users, particularly those in the CCM, IMT, and CSM CTAs.

Benefit to Warfighter: By providing easier access to HPC resources and software, and by
simplifying the construction of complex computational simulations and
problem solving environments, the Gateway system will improvide the
productivity and efficiency of DoD researchers.

Project Dependencies: This project is a joint effort of Florida State University and the
Ohio Supercomputer Center, together with the sponsoring centers, ASC
and ARL. OSC is primarily responsible for the development of the PSE
framework, while FSU is responsible for the middleware. This project
will primarily involve the PSE framework and its interface with the
middleware. The testbed environment (back-end computers) and
assistance with installation must be provided by ARL.

Members of the particular CTA teams will also have to play a
significant role in the design and development of new PSEs.

This project also assumes significant involvement of Derek Moses, the
part-time I/C on-site lead.

Risk Element: Portal technologies are a rapidly evolving area. The Gateway effort
is based on year-old best practices which are subject to be
superceeded by newer technologies. However
1) Gateway is intended to be an operational prototype system, and
keeping pace with the absolute latest technologies is at odds with
fielding a working system, therefore some compromise is
appropriate, and
2) Gateway has already successfully evolved from a custom-written
middleware to a commodity CORBA/XML-based middleware during its
lifetime, and unless there is a fundamental paradigm shift, we
expect it to be able to further evolve to take advantage of new
technologies as appropriate.

Institution Name:	Florida State University
Project Identifier:	ARL-CY5-IC--3
Project Title:	Extension of the Gateway System to Support New CTAs
POC:	Fox, Geoffrey C
Email:	gcf@cs.fsu.edu
Phone:	850 644 4587
Fax:	850 644 0098
CTA:	IC
Project Description:	Gateway a three-tier system designed to provide a seamless web-based environment for accessing a broad range of back-end HPC services including parallel computers, visualization systems, database systems, etc. By providing a middleware, consisting of well-defined services and application program interfaces (APIs), the broad range of back-end services can be presented at a level of abstraction which separates applications from many details of the back-end implementation. The front-end, or user interface, connects to the middleware using its abstractions and APIs, thus providing tremendous flexibility and generality to the front-end. Front-ends can take many forms depending on user needs. One example would be using the Gateway environment to develop a complex multi-physics/multi-scaling simulation by coupling of several HPC applications. Another example, which is the current focus of PET Gateway activities, is the development of domain-specific problem solving environments (PSEs) which provide a suite of tools under a unified interface designed to assist users in a given CTA or field with everything from input preparation to job submission and execution, to analysis of the results. The Gateway project, a collaboration between Florida State University (the team having moved from Syracuse University), the Ohio Supercomputer Center, ASC MSRC, and ARL MSRC, uses a CORBA and XML-based middleware and modern web technologies to provide this seamless environment. During Year 4, PET support has focused on the development of a CCM PSE (other types of front-ends are being developed with other funding) and on deployment of the system in testbed environments at both ASC and ARL. These testbeds provide user feedback and operational experience which is crucial to preparing the system wider deployment, possibly into production environments. An important part of our Year 5 activities will be aimed at supporting the ARL testbed and improving the Gateway middleware to address problems identified during testing. The other primary focus of our Year 5 work will be the expansion include PSEs for other areas. The CCM PSE serves as an exemplar of one common class of HPC use, where many long-running compute-intensive jobs are executed in batch mode, accompanied by input preparation and data analysis tasks which are, in comparison, computationally modest in their requirements. This is a large class of applications, which is why CCM was chosen as the initial target, but it is by no means the only way to interact with HPC systems. Therefore, despite conscious efforts to construct the PSE framework and middleware interface in a general fashion, there will inevitably be modifications required to accomodate other classes of HPC users. Therefore, during Year 5, we plan to collaborate with OSC and appropriate CTA teams to develop one or two additional PSEs targeting other classes of use in order to insure the appropriate level of generality in the PSE framework and the middleware. Such a step is necessary before we can consider opening up the Gateway system to application developers across the Modernization Program, inviting them to create PSEs for their fields. Our initial target will be the IMT CTA, which presents a very different model of HPC use. IMT often involves large volumes of data and requires flexibility in connecting the data with analysis tools. It is also more interactive than batch oriented. The ARL PET team has already developed a prototype secure web-database environment which will serve as a starting point for a more general IMT PSE. Our plan is to have an initial IMT PSE running on the ARL testbed and available to users by the end of Year 5, with intermediate deliveries of designs, prototypes, and limited pioneer user testing to be determined as the specifics of the new PSE are fleshed out. Once work on the IMT PSE is well underway, we may begin the development of a second new PSE. Currently we see CSM is particularly promising because it seems to offer a combination of feature of the CCM and IMT areas. Substantial batch-oriented computing combined with intensive analysis of large data sets thus produced. In particular, this field seems to offer an excellent opportunity for the integration of data mining tools. Work on a CSM PSE is not necessarily expected to be completed during Year 5, but should be sufficiently advanced to serve as a confirmation of the architectural modifications resulting from the IMT PSE work. This work, together with related activities during Year 5 at ASC will provide the basis for the development of training materials to introduce others to PSE development for the Gateway system. A workshop on this topic is part of our I/C Training proposal, and will help set the stage for opening up Gateway to the larger HPCMP community. Florida State's role in this work, as in the past Gateway work, will focus primarily on the middleware, while OSC will focus on the PSE framework. This project will help to refine the interface between these two tiers of the system, and so will require the close cooperation of the two organizations.
Project Objectives:	The Gateway system, with a CCM PSE, is being deployed in testbed environments at ASC and ARL, and will soon be open to pioneer CCM users. The development of specific new PSEs will help to insure that the PSE-middleware interface and middleware services are defined at the appropriate level of abstraction and with appropriate interfaces to accomodate the requirements of a broad range of PSEs. This is the essential next step as we move towards the ability to deploy Gateway as a production system at the MSRCs, and open up PSE development to the entire HPCMP community.
Deliverables:	1) On-going support and enhancement of the ARL MSRC Gateway testbed installation, together with ARL MSRC staff 2) Support for the development of new front-ends for the IMT and CSM CTAs (specific CTAs subject to revision in disccusion with PET management), in collaboration with the Ohio Supercomputer Center and appropriate CTA team members 3) Development, in collaboration with OSC and ASC, of training materials to help open the development of new PSEs for other CTAs
Customers/End Users:	HPCMP users, particularly those in the CCM, IMT, and CSM CTAs.
Benefit to Warfighter:	By providing easier access to HPC resources and software, and by simplifying the construction of complex computational simulations and problem solving environments, the Gateway system will improvide the productivity and efficiency of DoD researchers.
Project Dependencies:	This project is a joint effort of Florida State University and the Ohio Supercomputer Center, together with the sponsoring centers, ASC and ARL. OSC is primarily responsible for the development of the PSE framework, while FSU is responsible for the middleware. This project will primarily involve the PSE framework and its interface with the middleware. The testbed environment (back-end computers) and assistance with installation must be provided by ARL. Members of the particular CTA teams will also have to play a significant role in the design and development of new PSEs. This project also assumes significant involvement of Derek Moses, the part-time I/C on-site lead.
Risk Element:	Portal technologies are a rapidly evolving area. The Gateway effort is based on year-old best practices which are subject to be superceeded by newer technologies. However 1) Gateway is intended to be an operational prototype system, and keeping pace with the absolute latest technologies is at odds with fielding a working system, therefore some compromise is appropriate, and 2) Gateway has already successfully evolved from a custom-written middleware to a commodity CORBA/XML-based middleware during its lifetime, and unless there is a fundamental paradigm shift, we expect it to be able to further evolve to take advantage of new technologies as appropriate.