ASCI PSE/DisCom Mileposts Review Panel
Lawrence Livermore National Laboratory
Building 451, Room 1025
April 10-12, 2001

Participants:  

Panelists: Bob Lucas, Laurence Merkle, William Johnston, Charles Koelbel, Frederick Johnson, Charles Catlett, Roscoe Giles, Geoffrey Fox, James Browne, and Bill Carlson

DOE HQ: Bill Reed, Jose Munoz, Michael Underwood, Thuc Hoang, John van Rosendale and John Engel

Lab Personnel: David Nowak, John Gyllenhaal, Mark Gary, Mary Zosel, Art Hale, Jim Ang, Jeff Brown, Steve Turpin, Michel McCoy, Jeff Brown, Jo Sander, Mike Koszykowski, Bar Chan, Tom Pratt, Randall Rheinheimer, David Wiltzius, Curtis Janssen, Chris Chambreau, Randy Christensen, David Dannenberg, Karen Warren, George Richmond, John May, Steven Humphreys, Joseph Slavec, Francais Gygi, John Engle, Evi Dube, Scott Futral, John Naegle, Ray Berg, John Kelly, Rich Zwakenberg, Martha Ernest, Tyce McLarty, Buck Thompson, George Richmond, Linda Stanberry,, Terry Heidelberg, Jeff Vetter, John May, Richard Hedges, Brian Carnes, Steve Ashby, Bruce Griffing, Doug East, Jean Shuler, Phil Heermann, Steve Louis, Steve Tenbrink, and Tina Macaluso

ACTION ITEMS
Tuesday, April 10, 2001

•	Panel members asked that the following information be provided to them:
•	A single slide that defines technology goals for ASCI and then identifies what components are in place to meet the goals (requested by Johnston, agreed to by Zosel)
•	Answer to the question, what is the bandwidth for the White adapters? (requested by Carlson, agreed to by Louis)
•	The Panel presented the following questions to the PSE Leads with a request for additional information on Wednesday, April 11, 2001:
1.	What is the context of this set of PSE mileposts (in the space of codes that solve the overall problem)?
2.	What is the life cycle of these tools?  Will they be portable to Compaq?  Will they be portable to Linux?
3.	Who owns the SW development environment?  Who owns the actual development tools?  (frameworks as an example)
4.	Do these tools span the space of needed capabilities?  Will they be adequate for other ASCI codes?
5.	Are the objectives of these mileposts relevant to the future SW development problems of ASCI?
•	Mary Zosel agreed to determine why some of the presentation information could not be found by panelists on the CD provided before last week (requested by Fox).

Wednesday, April 12, 2001

•	Panel requested a first order breakdown of the DisCom2 budget with an emphasis on the costs associated with meeting the milepost.  (requested by Lucas, promised by Hale)
•	Panel requested an interactive running of TotalView as part of the classified DisCom2 demo (requested by Lucas, promised by Hale)
•	ASCI HQ agreed to provide the panel with a list of application “power users” for follow-ups (requested by Lucas, promised by Reed)

DETAILED DISCUSSION HIGHLIGHTS (Open Sessions Only)

Tuesday, April 10, 2001		PSE Presentations

Bill Reed, Opening Remarks
•	We have two important mileposts to review at this meeting.  First, let me begin with a welcome for all of you.  We appreciate your being here.
•	We have had Blue Ribbon reviews of all of ASCI but this is the first official review of a major milepost for PSE and DisCom2
•	We pay a great deal of attention to reports from review committees.  We will be listening very carefully to your conclusions and recommendations.

Jose’ Munoz, Opening Remarks
•	I would also like to thank the members of the committee, particularly Bob Lucas, for the time and effort you have agreed to put into this review
•	We want to hear what we have done well and what we have done not so well.
•	Feel free to be critical, tell us what we can do better and what we should do in the future
•	It is important to recognize that these are two separate reviews and that these are the first truly TriLab mileposts that are being reviewed
•	We have a great deal of material to go through and not a lot  of time

Bob Lucas, Introduction of Committee
•	Roundtable introductions (Lucas, Merkle, Johnston, Fox, Giles, Catlett, Browne, Johnson, Koelbel, Carlson)
•	This is a very crowded agenda so  we will need to manage our time carefully

Mary Zosel, Overview of Agenda
•	We have tried to build time into the agenda for question and answer sessions
•	We can also arrange one-on-one sessions for Thursday as needed
•	We are going to begin with some background talks to provide a general picture of the White platform and then we will go into the details of the milestone
•	Our PSE milepost was completed in the fall quarter and ended in December.  We have worked since then on the materials that you see today.  The DisCom2 personnel just completed their milepost last week so they have not had the past three months to assemble materials.

Art Hale, Opening Remarks
•	I would also like to thank the committee members for taking  the time to look at what we have done
•	DisCom2 personnel will be largely absent today but will join us tonight for the social  function and will make presentations tomorrow

Jeff Brown, PSE Overview
•	My goal is to begin with a quick overview of the TriLab PSE Program
•	PSE leads are myself, Steve Louis, Mike Koszykowski and Thuc Hoang at HQ
•	PSE focuses on providing the necessary software required for successful applications
•	PSE is one component of Simulation and Computer Science (S&CS)
•	Academic staff refer to domain-specific PSE(s) but the ASCI PSE is much broader in scope
•	There are three basic thrusts:
•	Algorithms scale and adapt to  platform structure
•	Application development (deploying debuggers, compilers, performance analysis tools) using common environments
•	Platforms usable at scale (scalable RTS, IO, storage, OS, etc)
•	PSE is driven by the needs of the applications and platforms
•	PSE works with other programs to make the platforms usable
•	There is an increasing gap between commercial drivers and ASCI needs.  Future platforms will leverage small-scale commercial building blocks that must work as a system
•	We are looking seriously at Linux as a possible key to our software future
Carlson – There is some relationship that this program must have with the individual vendors of the platforms.  You are describing some low-level components to support these systems
Louis – I will describe that somewhat in my presentations.
•	PSE must push these technologies to get what we need (at the scale we need)
Koelbel –What do you see for the future with Linux?
•	We are interested in Linux because we want to gain more local control over our software.  The SGI work does not all carry forward to the Compaq, for example, so we would like to invest in some SW infrastructure that is leveraged in multi-platform. 
Johnston – Do these bullets characterize your definition of a PSE?
•	No, they represent the critical pieces of a software system that we must have
•	The scope of PSE is the user-level software that enables applications to work.  This includes solvers, frameworks and performance analysis.  The PSE organizational structure has four components: 
•	ASCI Simulation Development Environment, 
•	Data Transfer and Storage (pipes and bytes), 
•	Distributed Systems and  
•	Management and Integration
Browne – Who does the data analysis software?
•	That is VIEWS.
Browne – I saw a goals and requirements statement for this panel when I was on the Blue Ribbon Panel (the first panel)
Louis – At the time of the first BRP, VIEWS was actually a part of PSE.  NEWS included the hardware component
Browne – Somewhere there is a goals and requirements statement that we looked at three years ago.
Crawford – It was the Kiviat diagram that we used to define what was needed, I believe
Browne – This was a roadmap that had a tree structure.  There are many ways to determine how well you have done.  It would be helpful to look at what you planned to do at the start.
Zosel – We are reviewing the milepost activity as opposed to overall PSE
•	We do not necessarily write all of the software
Lucas – On page 6 of the milepost report, there are several paragraphs defining the milepost
Crawford –There are specific metrics against which we are to measure the milepost.  Mary will discuss that in her talk.  What Jeff is discussing now is broad background.  
Koelbel – I am now confused with where PSE ends and where VIEWS and DisCom2 begin
•	VIEWS has to do with visualization.  It used to be part of PSE but is no longer.  DisCom2 focuses on computing at a distance (and distributed computing) – this includes networking, etc.  PathForward funds companies outside the laboratories for enabling technologies such as interconnects that supplement what the platform vendors provide
Johnston – It would be useful to have a single slide that defines the technology goals for ASCI and then identifies what components are in place to meet those goals
Zosel – We will bring that back to you tomorrow morning.  ****** ACTION ITEM
Fox – POOMA is a development framework?
•	Yes, and it is not part of this milepost
Johnson – Does PSE include all of the system tools?
•	Yes, it is all of the local system software that is leveraged with the platform.  We are responsible for making certain that all of the needs of the application are met.
Lucas – How is the networking in PSE different from DisCom2?
Koszykowski – DisCom2 is also a new program.  When it first evolved, DisCom2 focused on the encryptor, while PSE continued to do things like the WAN.  PSE still performs some R&D that DisCom2 has not picked up.
•	Our focus for next year is more on local computing than it has been in the past

Steve Louis, ASCI White 12 TeraOPS Platform Overview
•	This is the second overview talk.  My goal is to set the stage for the hardware environment for  the PSE milepost
•	PSE funds the integration of ASCI platform hardware and software in cooperation with LLNL production computing
•	All of the hardware for ASCI White was delivered in September 2000 and was used (with Frost and Snow machines) for the PSE milepost.
•	White was initially used as two machines (Snow and Frost).  Snow was used for functionality primarily, White for scalability.  The two machines were merged in February.  White, Frost and Snow use the same software but Snow has slower NH-1 nodes
•	White peak performance is 12.3 trillion ops per second (overhead with system details)
•	Data visualization and data mining can be performed on the 32 VIEWS nodes
•	We are in the progress of upgrading all of the nodes to 16GB memory per node
Lucas – Are there 16 GB of memory per node on ASCI Red?
Crawford – No.  The issue at Sandia is the total memory required to solve the problem
Brown – There are some codes that require a single memory space within an SMP
Nowak - Power requirements are measured in megawatts (in the low integers) and depend on how much memory is on the machine.  The actual usage is less than IBM is claiming it would be
Johnston – Is the memory configuration based on requirements?  Bill Kramer (NERSC) ran into problems with the need for enormous buffers that doubled or tripled the amount of memory needed per node.
Nowak – It is driven by the size of the code and the size of the binary
Brown –This is somewhat of an implementation-specific issue
Carlson –What is the BW for each adapter?
Louis – It is 250 or 500 MB/s – I will get that for you ****ACTION ITEM
Koelbel –How were the GPFS nodes and the VIEWS nodes split up?
•	There were no VIEWS nodes at the time.  The GPFS nodes were split 8 and 8.
•	Mohonk is the current software running.  We will upgrade to Mohonk-2
Lucas – Did you originally ask for double double (two adaptors and four planes)?
•	Yes.  We are doing more adaptors but not more planes
•	The software on White comes from multiple sources: IBM, third-party developers, LLNL
•	PSE usually had a large influence on specification, development, testing
•	There are five current programmatic drivers: DisCom2 Milepost, Sandia Engineering milestone, LLNL milestone A, LLNL and LANL Milestone B and DSW.  The machine is fully utilized and is more stable than we thought it would be by this time
•	The White Policy Committee meets periodically to set access.  We expect to meet all of the mileposts (according to the revised schedule)

Mary Zosel, ASCI PSE Milepost Overview
•	Our goal (back in 1997) was to demonstrate a truly usable environment on ASCI White with software working on scale that promotes portability
•	PSE was involved in the software in different ways (specified in Appendix D)
Fox – POOMA is not in the milepost. Does that mean that no one wanted to use it?  I am confused between science and other requirements.
•	The application personnel at the three labs selected the codes.  POOMA is not slated for a milepost run
Brown – Blanca is alive and well at LANL and using POOMA.  It is very much supported and an important piece of software but was not selected for this exercise.
Nowak – My impression is that POOMA is not widely used at LANL
Fox –Frost or Snow is a fully utilizable machine, according to Mary’s statement.  This may be true for the milepost-selected codes but may not be true for all ASCI codes
Brown – Each lab identified key codes to be drivers.  These sets of codes provided the requirements.  Blanca and POOMA were not selected for this milepost.
Fox –There should be a plan stating that we need this code at a given time
Nowak – There is such a plan.  POOMA was not selected at this time.
Koelbel – When were the codes to be used for this milepost chosen? Was there any long term planning?  Is there any guarantee that all of the codes will be tested?
•	Between July and August of last year in conjunction with our applications at the labs
•	PSE is not responsible for porting codes to the machines.  We provide the environment in which those codes run.  POOMA is a split responsibility between apps and PSE at LANL
Johnston – What fraction of the overall problem does the tested codes represent and why were the others left out?
Christensen – The milepost codes LLNL selected were those we expected to be using soon on the White system for applications and DSW.  Those became our targets.  The milepost was not to be ready for every possible code but to be able to support our immediate workload
•	We also are not talking about solvers right now and this leaves many codes out
Brown – We picked OpenMP, Fortran, and C based codes
Browne – The issue is whether you have picked a representative sample and what capabilities are there as a result of meeting the milepost.  We had to pick a finite set.
Carlson –Is POOMA not ready?
•	There are many applications that are not included here.  Blanca was not included in this milepost because the POOMA team at LANL is in transition.  
Brown – We are rebuilding the POOMA effort (it is alive and well).
•	We had 60 detailed tasks associated with this milepost.  We shook out compilers, development tools, we dealt with machine access issues, verified I/O and archival systems as usable and ensured that application codes, I/O and tools can function at scale
•	In general, there are many ways the IBM platform will be better
•	We will ask the review team if you believe that we met this milepost definition
Koelbel – Who is the “we” that has learned these things?
•	The LLNL Second Aid Team passes questions and queries on to the lab PSE team.  The people you are going to hear from are on the Second Aid Team
Johnson – It was hard for me to assess how much work went into this. How close is this capability now to being routine (as represented in the milepost)?
•	There are parts of running on the big machines that I think are close to being routine and there are other parts that still need work.  Things are becoming routine but are not there yet.
Koelbel – The Second Aid Team has good knowledge but how is what they are learning being shared with users?
•	Our code development teams include computer scientists who are very good and they integrate what we are learning onto the team
Koelbel –What are the knowledge transfer processes you use?
Louis – Hotlines, reflectors, distribution of everything learned to all White users
•	Part of what you will hear tomorrow will be about distribution of knowledge.  Jean Shuler will talk about user services aspects
•	Our code test objectives:
•	Run small-scale tests on Snow of all codes
•	Run one code from each lab at 1500+ processors
•	Did we satisfy the PSE milepost definition?
•	Did we meet our objective of showing the ASCI Initial Simulation Environment on White?

Mary Zosel, LLNL Application Runs for Functionality and Scale
•	Our objectives were to demonstrate compiler/build software, to demonstrate the OpenMP option, to perform functionality testing of ALE3D, Ardra, Jeep, to scale test ALE3D and to supply codes for tools testing
•	These codes were already running on the platform
•	Accomplishments: ALE3D, Ardra, Jeep runs on Snow; I/O recorded and archived for each, ALE3D run  on White
Browne –Why was OpenMP not good at low sources of domains?
•	When you have fully decomposed your code at low numbers of domains, you encounter the usual suspects.  Shared memory brings new dimensions.  Typically when a code is decomposed (ALE3D is a special case) there is one domain per MPI task per CPU.  ALE3D over decomposes and puts a pile of domains onto one MPI task.  This was actually a real computation on a real mesh.
Koelbel – A code that was perfectly scaling would be a horizontal line?  Answer: Yes
•	The code would not handle this mesh at the 1500+ domains.
Johnston – Does OpenMP only handle local memory?  Answer: Yes
Koelbel – Do Ardra and Jeep also use OpenMP?  Answer: Ardra is mixed, Jeep is not.  Jeep is a good tool for breaking hardware
Merkle –What is the issue with 15 versus 16 processors?
•	One theory is that giving a processor just for running extra threads may help.  We have some suspicions about IBM scheduling tasks to nodes.  

Jeff Brown, LANL Briefing
•	Our objectives were to get LANL codes up and running (raise the visibility and work LANL issues ensure supporting tools, demonstrate PSE components of distance computing) and establish a process for the LANL 30T
•	If we were to initiate this milepost today, we would include a POOMA code.  We have a contract to provide POOMA (there is a subcontract with the POOMA team that left LANL).  We are also rebuilding a smaller POOMA effort at LANL.  The company with our contract (Code Sorcery) wants to offer services in high performance computing applications
•	PSE’s role was to demonstrate the execution, I/O and storage functionality for the codes in the LANL suite, code scalability for SAGE, tool functionality and scalability
Fox –What is Crestone?
•	The code teams at LANL are named after 14,000-foot mountain peaks in Colorado.  Crestone is one of the teams.
•	As a result of this effort, LANL staff became familiar with the White compute environment and established a viable TriLab process for standing up the initial development environment that we plan to repeat for the 30T
•	We received excellent support from the LLNL hotline folks
•	Details about  the codes in the LANL software suite (see overhead for specs)
•	UPS stands for Unified Parallel Software that has gather/scatter capabilities.  UPS is used in Eolus, Antero and other teams at LANL.  UPS was designed, developed and built at LANL
Louis – Bob was able to get more than 1 million CPU hours on the machines here at LLNL
Johnson – So this code was running on the big SGI cluster at LANL and you ported it to IBM to run on White?  Answer: Yes.
•	Truchas code ran successfully on all 30 test problems using optimized compile and multiple CPUs of Blue
•	We accelerated deployment of the distributed memory capability on White and we accelerated working OpenMP issues on AIX.  We ported the Unified Parallel Software library to White
•	Perfect scalability would result in a flat line (same amount of grind time).
Browne –What additional complexities would have entered with AMR?
•	In fact, one of the curves on this graph has full physics and AMR.  The idea is that we can run at scale and will get the right answer.
Carlson – Is it acceptable to say my scaled code has zero performance?   We are supposed to demonstrate scaled running of an application.  What does that mean?
Zosel- The code teams are responsible for showing how their codes scale. We are responsible for running MPI for them.  The teams are now looking at changes that may be needed.  
•	Our goal is to provide the system software tools.  The apps teams tune the codes.
Koelbel – How do you distinguish between when MPI is not working and when the code is using MPI poorly?
Zosel – Jeff Vetter will address that question this afternoon.
•	The number of results per unit time is the definition for grind time.
•	We demonstrated that the TotalView user interface works at a distance over the WAN on the secure network
Catlett –Did you not think it would work over a distance?
•	We thought there could be latency and firewall issues and found that not to be the case

Ray Berg, Code Activities at Sandia
•	We picked codes with diverse features and looked to see how well they performed in key areas such as security, I/O file system interaction, compilers and the like.
•	We were looking to test the PSE environment characteristics, particularly how well the codes could be built, and use of libraries, compilers and tools.
•	CTH is an Eulerian shock physics code that was converted to be built on White and ran “out of the box” on Blue and White.  CTH has a well-developed user community (250 users). This code is export controlled and is a legacy code using MPI (has been converted).
Hale – It is a structured grid but is not adaptive.
•	ALEGRA MP activity identified required compiler switch changes on the Blue Pacific machine builds.  ALEGRA experiments identified needed upgrades in GPFS in analysis of December tests.  There is a performance issue (8 node White runs are slower than 8 node ASCI Red runs) that needs further investigation.  This is being looked at now.  There have been no scaling tests on White run yet.

Curtis Janssen, MPQC Milepost Activity
•	The objective was to compile and run the Massively Parallel Quantum Chemistry program (C++, Pthreads, multi-threaded MPI) and to test tools at scale (TotalView, Gprof, MPI Perf, ZeroFault) and to test I/O (HPSS, GPFS)
•	Tested successfully a tool of each type including tests at scale (1500+ CPUs).  Tested successfully GPFS and HPSS and exposed several bugs that were subsequently fixed.
•	We verified that the tools were functional, usable at scale and fixed some bugs.
Carlson – This is an embarrassingly parallel application?  
•	The global memory is available through MPI and threads.  The idea was to hide the latency of the communication.  We are running 16 threads per node.  We are competing against IBM MPI library.
•	Review of tool functionality: ZeroFault
Carlson – You have run this on many systems.  Are there systems on which scaling issues exist?
•	We have not run as large before.  The scaling issues primarily exist on smaller data sets, which is probably a combination of things.  I would say this is more latency limited than bandwidth limited.
Giles – You identified compiler bugs and linkage bugs.  Using PSE helped you resolve those issues.  Is it your intention to scale this to actual production environment?
•	I frequently send this information to the LC hotline and they are very good at distributing it.

Mary Zosel, Introduction to Milepost Tools Testing
•	We are now going to discuss the tools that we ran on the codes
Browne – How did you decide what you wanted in the way of tools?
•	We picked basic bread and butter tools people need to do development.  The most requested tools are memory tools.  We wanted to show that some things work.
•	We are not a software development house per se.  Our goal is to provide the best development environment
Koelbel – Were the tools used remotely or locally?
•	LANL was remote; Sandia was across the street (is that local?)
•	Some highlights:
•	It takes 140+ compiler versions to give complete support required
•	The memory tools are a big addition to the environment
•	The ASCI-drive PathForward additions to Vampir are impressive – KAI/Pallas is already making big improvements
•	TotalView –critical to our users –functionality is constantly improving
•	Profiling in threaded environment is tricky business
Johnson – ASCI has made major investments in TotalView.  Are there ASCI capabilities that are not commercially available?
•	They will be introduced into the Etnus product line a short time down the road.  You could talk with the Etnus people and accelerate the work if you have some funding.  Etnus has gotten a little tight about whom they will give their source to.
•	The opportunity to test tools at scale was very valuable for the tool developers
•	We are finding some areas where our trend analysis can be additionally helpful
•	We found bugs early (before the users did).  People at all three labs know how to use the tools.  New tools are  in progress
Fox – Is MPI a tool?
•	We included the runtime libraries as tools.  We can provide more details if you have more specific questions.
Fox – I am a little confused by the scaling studies discussed this morning.
Brown – It was not an objective of the milepost to actually perform the analysis.  I believe that we are able to point the SAGE people to performance reasons for that curve
Carlson – This committee has been asked to confirm that something has been done but we do not know what it is. Are the scaling numbers sufficient?
•	Yes, the requirement was for the code to run on 1500 CPUs.  There are no performance criteria
•	We did not parallelize these codes. 
Brown – This milepost is focused on the initial development environment.  As the environment evolves, the performance issues will be addressed as part of the maturing process.  We need to make clear that you are not reviewing performance.  What you are referring to is an application milepost that has its own specs with respect to size and quality of physics results.
•	Having the I/O that works is critical to the applications milepost that you are asking about.
Louis –Please look on page 27 of the milepost report.  Note the mileposts up the curve from where we are today.  
•	The application milestones run for weeks and consume the machines.
Koelbel – Surely there are implications from the PSE performance for these other mileposts
•	Yes and the biggest implications are for the I/O side of the house.  Whether a compiler is fast or slow may have little to do with it.
Carlson – Whether MPI scales is crucial.
•	PSE is responsible for ensuring that we have an MPI that runs.  Some codes are doing well, while others are having problems.
Brown – The focus of this milepost is on the initial development environment.  We have only demonstrated that a code can run over a large number of CPUs.  We have not addressed performance.
Carlson –The words say run at scale, sufficient to meet those objectives.  Is this the case?
•	I believe the answer is yes.  I think the code developers do not see MPI as a major problem.
Crawford – The question is do the tools run on 1500 processors so that when the apps scale up their runs, the tools will work?
Browne –Do the tools run or can the users effectively use them?
•	The latter.  Do the tools run so the users can effectively use them on 1500 processors?
Crawford – The apps folks don’t have to worry about whether the compiler is right, getting MPI bugs and similar issues.
Koelbel –We talk about these tools as if there is a binary right and wrong and that is misleading.  Testing can only show the presence of bugs (not their absence).
•	We found bugs.  And, we found a work around for every problem found to date.
Koelbel – Performance does matter.  Having a correct Turing machine can prove existence but does little for performance.
Browne –ASCI mileposts are going to go on after most of us are dead and gone.
Crawford – There are many more PSE milestones – on this curve are only those that bring critical things together.  There is a 600-page PSE implementation plan for a single year.  What you see on this curve are major things that the program has accomplished.  We now call mileposts, Level I milestones.
•	People are not beating down the walls to say that the MPI is not performing.  The basic stuff is working.  
Brown – The SAGE curve simply represents a point in time test that shows that the code can run across 1500 CPUs.
Giles – We are talking about two things.  One is what we are here to judge with respect to PSE and this is a deliberately narrow view.  There is a separate question as to whether the milepost makes sense with respect to contributing to the overall ASCI goal.  The deep question is whether it is possible for everyone in ASCI to meet all of their mileposts and still have nothing work.
Lucas – We would like a slide that specifically defines what we are looking for that we can review after lunch
•	We will do that.  The 60-item checklist is also helpful for understanding the milepost.

John Engle, Compiler/Language Support for Tri-Lab ASCI Apps
•	Our objective was to confirm that compilers work for the PSE milepost codes on White (IBM and KAI compilers for f77, 190, C, C++ with OpenMP and MPI support available)
•	OpenMP is the preferred mode of operation for shared memory.  We have been working with the vendors on OpenMP and have had some successes.  We had some trouble debugging OpenMP, for example, and specified some options to IBM to help with the debugging that IBM subsequently implemented (with a few dollars from us).  From what I am hearing from others in the industry, this puts IBM ahead in debugging so we are pushing things forward.
•	The codes that would not compile at the start of the exercise now compile.  We identified and fixed both compiler problems and language/user problems and we found correct compiler options to use in specific cases. 
•	We reported the problems we found to the vendors and have gotten some fixes back already.
•	Our experience is that the ASCI codes tend to push the compilers more than the other customers these vendors have
Giles – If you look at the 10 problems (on the process tree), only 2 of them represent the user doing something wrong.  This is amazing.
•	Remember that these codes were already running on other platforms
Koelbel – Even so, having only 7 compiler problems is very small (less than one per code)
•	If we subtract out the LLNL codes, there are more than one per code
Zosel –User skating on the edge of C++ standards and f90 issues were the two major areas.
Browne - Let us recognize that C++ is a terrible language to compile.  Agreed.
•	There were three cases where there was an initial problem that had been discovered and fixed so they did not appear when we ran the latest beta release.
•	Compiler version control is a challenge.  We have a standard path that we take the compilers through.  We have multiple vendors, several languages, different versions plus IBM use of compiler names results in 149 different compiler names
Browne – Some compilers don’t work with some libraries.  I had a problem with C++ compilers with standard template libraries.  Is that a factor here?
•	Not directly (with respect to the names).  The IBM C++ library will link to their version of the STL.  I maintain a web page to define what versions are tied to what compilers.
Zosel – There are some restrictions and sometimes the libraries cannot be mixed and matched.  The best solutions for addressing those problems are yet to be shaken out.
Giles – At some level, the object headers know what file they came from.  Can you track your versions over time?
•	Most of these names are scripts that lead to a particular configuration file so the information is buried in there.  The data describing which compiler compiled something does not tend to be in the object file (unfortunately).  We have spoken to the vendors about this.

Chris Chambreau, Profiling Tools Introduction
•	Our objectives were to evaluate the effectiveness of existing profiling fools (components that gather the profiling data including AIX OS, compilers, gprof and xprofiler)
•	The profiling tool is not a debugging tool.  Our intent was to display that it was available and functional for the developers.
•	We encountered multiple problems associated with scalability and functionality.  We addressed the problems through fixes, recommended use and new tool development
Koelbel –What are you going to do about the GUI interface clutter problem?
•	We have talked with IBM about the problem but it is not fixed yet.
Carlson - Gprof was the only issue you found associated with scalability?  
•	Essentially, yes.
Carlson – If profiling shows a difference between one thread and another thread, would xprofiler show you that distinction?
•	The data gathering mechanism does not distinguish between threads.  Using the call tree, you can look at the min and max time associated with the function.  We have put together a script that essentially parallelizes the profiling and creates a load balancing.
•	Generally, compile and link profiling examined through Xprofiler is most useful and convenient.
•	In summary we were able to generate profiling for 5 of the PSE apps (3 at scale)
Q - Are DPCL and Benchmarker open source?
•	DPCL is but Benchmarker is proprietary
Johnston – Once you get past the easy issue you are left with the fundamental issue (GUI is ineffective for large apps).
•	Our approach was to essentially disregard the GUI and to use the call graph applications and the flat profiler report (along with other data provided by x profiler).  We found that the text reports are what people are using.
Koelbel – Do you find the text reports to be useful?
•	If you are looking to see what pieces of the code are taking the greatest percentage of time, that is what you have.
Zosel – We are not all that excited about the profiling tools.

John Gyllenhaal, Detecting Memory Problems on IBM Power3
•	Our objective is to provide one tool that could be used to do memory corruption and leak detection on the IBM Power3 platform on the new White machine (for all code languages)
•	We developed techniques to make a few existing and mostly serial IBM-compatible tools work in a parallel environment
•	We were pleasantly surprised with ZeroFault (developed by Kernel Group out of Austin, TX).  (their real market is NT servers). This is a heavyweight detection of memory problems that uses a normal executable, is fast and was developed using a C compiler on a single node.  
•	Although we had ZeroFault installed for a while, it was rarely used since it didn’t work with parallel applications.  We built a “wrapper script” to trick ZeroFault into thinking each individual task is a serial process
•	We used ZeroFault to analyze Ardra, Jeep, Truchas, Sage and MPQC and found problems that were preventing milepost runs.  We exposed the problems, found workarounds and worked with IBM, Kernel Group and KAI to resolve problems
•	Users can now detect memory problems on the IBM system at scale.  We have seen rapid acceptance from the users and have a wish list of things they would also like to see ZeroFault be able to do
Lucas –What is POE?  Answer: The IBM parallel operating environment
•	We are also looking at other memory tools, such as Great Circle, which was used to detect leaks in ALE3D.  We looked at the IBM memory tools in Mohonk.  
Johnston – This provides you with a way to look at memory requirements that grow without bounds?  Answer: Yes.

Karen Warren, TotalView
•	The objective was to test TotalView’s functionality on codes that use a variety of compilers and various parallel programming paradigms
•	PSE has played a major role in the development of TotalView by Etnus, particularly the scalability and adding features our users need.  We beta test new TotalView releases.
•	We demonstrated both functionality and scalability.  TotalView enables ASCI code developers to get their codes running on ASCI machines.
Lucas –Why 1024 cpus for Partisn?  
Zosel – We did not have the time on the platform to run more.
•	The quotes presented here came from every ASCI code team last Fall
Carlson –TotalView is not without its faults.
Zosel – The major influence is positive.
•	PSE prototyped features are now part of the Etnus’ TotalView release: slicing, filtering, sorting, statistics, call tree, message queue graph, memory utilization, attaching to subset of processes, structured array member display
Browne – What is the overhead associated with running TotalView?
•	It depends on the size of the code that you are running.
Zwakenberg – It has the capability to set eval points and that could perturb the code
Brown – You have to compile for debug.  You have limited capability to debug optimized code.
•	PSE prototyped features not yet part of Etnus TotalView: vector subscripts, global format and precision setting, history window, data comparison, root window and data window condensing
•	Features in TotalView added via ASCI PathForward funding: OpenMP (KAI and IBM support), command line interface and scripting support, IBM 64-bit support, enhanced thread functionality, enhanced job launch and execution, data watchpoints, process thread set groupings, support for C++ STL classes, parallelization enhancements to the expression system
Fox – What guarantee is there that Etnus will continue to do this?
Brown – When ASCI started this project, our funding represented a significant portion of Etnus’s revenue.  We are down to 20% of their total revenue now.
Johnson – Is there a well defined interface between the OS and TotalView
Zosel – We must work with the platform vendors on a case by case basis to make that happen.  Etnus is good about defining APIs but IBM hates to take APIs from anyone.  It has been a challenge.  I would like to remind you that the report you have been provided that been classified as sensitive unclassified information and should be treated carefully.

Demos over lunch: Memory and TotalView

Evi Dube, Scott Futral, Sample Demo and PSE
•	We use TotalView heavily and also ZeroFault
•	Most of our solvers are in C++
•	The problem is a small piece involving 20K elements and three materials.  The problem is run with slip surfaces between the HE and the metal.  We decomposed into 88 domains.  To scale up to 30+ M elements, we increase zoning and add other parts to make a sphere.
Browne – Does the size of the array affect how you use the tools?
•	Yes, the tools become even more important with large arrays.  We can sort by actual value (or by index) and can slice a section of data that we want to look at closely.  We can sort by descending order, for example.
•	The TotalView people worked with us to find out what we wanted to be able to do with these tools.
Browne – Does the tool perform at the same rate if you have thousands of processors?
•	There is an overhead as the number of processors increases
Browne – This is intrinsically – at best – a linear operation.  
Zosel –You can selectively look at a subset.
•	You can narrow the scope to make it more interactive
Carlson – The issue I would have for running at large scale is stepping or doing a laminated display when you wanted to do something in a coordinated fashion at large scale.  Have you tried this?
Zwakenberg – We did that for a 96-process job on White for the milepost job.  TotalView has to launch a daemon for each process.  That went pretty fast.
Carlson – If I wanted to go to 1500 processors?
Zwakenberg – Right now it starts get bad around 512 processors.
Zosel – Startup was the first killer and step was the next.  They have improved both but it is still a challenge with large numbers of nodes.
Carlson – We are trying to assess usability of the system.  20 seconds may be fine, 20 minutes would not be.  
Giles – What kinds of bugs do you expect to see at scale that you do not see now?  Answer: Localized in space and time anomalies
•	Hitting limits in memory, deadlock situations.
Dube – We also are running slower at scale so it can take longer to crash. We run into IBM buffer problems at large scale.  The biggest thing for us has been the memory leaks/memory corruption.  

John Gyllenhaal, Memory Tools Demo
•	I use a “Hello World” code and I have it do terrible things to memory.
Browne – This is the kind of code I see every day.  How would this be different if you had 1000 MPIs?
•	You are only going to get data on one particular task. We purchased licenses to run up to 64 things in parallel.
Giles – Have you had scalable licensing discussions with these people?
Zosel –Actually this is a way that we can continue to provide finding to the company.
Browne –Does the Power3 have an undefined zero value?  Answer: No, if you read from null you get zero
•	I maintain a list of things that we learn and have fixed and I provide the list routinely to our users
•	Demo of Great Circle tool (replace Malloc with own version)  

Mary Zosel, Review of Panel Assignment
•	Review of text on pages 24-25, Milepost Report
Giles – This means that things work and don’t break.
Browne – This has nothing to do with running to scale?  Answer: Not yet.
Merkle –What is meant by performance for the tools?
•	Performance of MPI, OpenMP threads – this is a performance test suite
Catlett –Run a set of PSE performance and functionality tests would be better wording.
Koelbel –If you were able to run the tests and they showed poorer performance on the new machine (compared to the old), that would still be a success?  Answer: Yes.  Whether we get 2MB or 20MB/s – we do not own the code.
Carlson – What do you want us to judge with respect to: “At least one ASCI related code from each laboratory will be built and run at scale sufficient to meet the CY00 milepost requirement of 1500 processors”?  We cross out the NA-2.1 and NN-1.1?  Answer: Yes.  That is sufficient.
Koelbel –We are testing the speedometer – we have no performance target?
Giles – We want to see that the speedometer is attached and works.
Louis – We did not put a specific target of data transfer rate to meet.

John May, Using Hardware Performance Counters
•	The goal is to collect a performance profile that gives information about the relative load balance across the tasks and to create performance tuning tools 
•	We developed a multiplexing interface to do  time slicing
Johnston – Are these events random or can you filter them?
•	You tell the CPU what event to count 
•	We invented the MPX library that shows hardware counter data for single process or for parallel jobs.  We measured and exhibited cache utilization and load balance on ALE3D, Ardra, Jeep and Sage.
•	Users can now measure L1 cache utilization directly on IBM systems (we found a large difference in cache performance between 604e (on Blue) and Power3 (White)
•	There is a HW counter available now but it is not user-friendly and the vendor interfaces are not portable
Johnson – How do you determine the region to instrument?
•	Right now the user puts in calls
Johnston – You could potentially wipe out the cache and end up on another processor?
•	IBM handled that at our request
Zosel – On the SMPs, CPU-only counters were useless so we told them we needed process counters down to the thread level
Johnston – On the heavily computational-bound codes, the process becomes a candidate for reprocessing.  Do the data reflect the fact that the process may have switched?
Giles – Is there a circumstance where you might want to stop migration?  With scaling to large system you are probably looking for 25% effects.  Do you see migration to shared memory at that level or is that not an issue?
Zosel – If the CPU determines to preferentially bind two processes to the same processor, then load balance is substantially affected.
Carlson – What you are describing is closely tied to features of the IBM system.  How do you account for the fact that the next machine will be a Compaq machine?  Where is the tradeoff between making the IBM machine the best it can be and assuring portability for future platforms?
•	I could have written this code on top of the IBM library instead of the PAPI library.  The PAPI library developers are trying to fix the problems associated with working on the IBM library.  There is a tradeoff but the preference is to get something working.
•	DPCL is another can of worms.  It has recently been released as open source by IBM (something we pushed them to do).  
Zosel – The motivation behind the Tool Gear is to increase our potential to go to other platforms while working on tools for IBM
Merkle – Do I get data from different processors on your graphical interface?
•	That is work in progress

Linda Stanberry, Run Time System Performance Testing
•	The objective was to track run-time system performance in HW changes (from Blue to White), implementations and tuning, Users’ MPI scalability perception
•	We measured performance of a full range of key run time systems: MPI, Pthreads and OpenMP.  We ran on Snow, White, Frost and Blue.  We found some performance problems and directed IBM in what was needed to fix them.
Fox – Just a small item but these things are not on the CD or the web site.
Zosel – It is probably a permissions issue and we will fix it.
Koelbel – Do you have any idea what the variance is on the threads timing?
•	No, we are discussing this with IBM now
Zosel – That is a good point.  
•	We are asking IBM if their model for thread scaling is more directed towards client server models than high performance applications
Browne – One of the things that makes threads more efficient is scheduling them on a run completion basis
•	The OS does not have that option.  You can fake it by setting the priority of the thread so it will not be preempted
•	We developed a test suite to compare the IBM, the MPICH MPIs and other implementation.  Our test suite is written to be portable.  We were able to provide our users with advice for tuning their codes. Sphinx is openly available right now.
Koelbel – I think there would be much interest in this in the external community.  Have you had many requests from users to help tune their codes?  Answer: Daily.  Have you looked into whether there is a correlation between these tests and the complexity of the user codes?
•	Yes and there is a correlation.  
May – We had the Pthreads tests working on DECs, SGI and IBM

Jeff Vetter, mpiP: Lightweight and Scalable Performance Analysis
•	The objective is to gather timing information for MPI at production ASCI scale (hundreds of MPI call sites and thousands of MPI tasks).  We also wanted to provide an alternative to tracing
•	We tested functionality and scalability on applications on ASCI AIX platforms
•	mpiP was successful form 1 to 1536 tasks.  LLNL, LANL, SNL and Caltech have requested mpiP ports to their platforms.  These ideas are being incorporated into KAI/Pallas PathForward products.
•	We use the MPI profiling layer so it is very portable (written in C).
•	mpiP is a straightforward, scalable tool for MPI performance analysis
Carlson – Scaling is based on 1500 files?
•	There is a line for each – you can determine which call site is creating your scalability problems.  
Zosel – The ASCI apps people told us they wanted this tool immediately when we presented it to them.

Jeff Vetter, Pallas Vampir, MPI Performance
•	Vampir provides detailed information about MPI.  This tool is more heavy duty than mpiP.  Our role is to provide users with guidance on using these tools.  PSE is accelerating KAI/Pallas products through an ASCI PathForward contract
•	We performed MPI experiments on moderate size applications (32-64 tasks).  We worked with all applications and found MPI issues in several codes.
Fox – Why is the data on the CD for Vampir not consistent with what we heard this morning?
Janssen – One does not get the performance with small data sets.

Jeff Vetter, Demonstration of trace file from Ardra
•	Vampir does not yet give breakdown by call site 
Hoang – Perhaps you should mention that this is a relatively new project
•	That is true.  The contract is with Cook and Associates (now Intel) and Pallas (a German company).  We have another round of major deliverables due shortly.
Zosel – We are initially putting this on IBM and will then port to Cplant and Linux.
•	The global time line loses some of its usefulness as you scale up but you can scale it per node and that helps
Zosel – OpenMP performance is now integrated with MPI performance
Lucas – Will the interface look the same?  Are you working with Pallas in a manner similar to how LLNL is working with Etnus?
•	The mpiP stuff is different from what they are used to.
Zosel – We started working with Etnus five years ago.  We are just getting started on this one.
•	We are implementing Art Maloney’s approaches as part of technology transfer into what we are developing
Carlson – Does this imply that when you are doing MPI you are going single threaded?
Zosel – It is probably safer to have a single CPU managing all of your MPI because you end up with locking if you use MPI for multiple threads simultaneously
Johnston – OpenMP effectively shut down during MPI.  This morning we saw a graph suggesting that OpenMP kept running during MPI.  Is this a function of the code?  Answer: Yes.
Zosel – A number of code folks tell us they are embedding Vampir into their codes so it is clearly catching on.
Brown – PathForward gives us a way to impact what Pallas does with this product.

Steve Louis, I/O Introduction
•	We are talking about (1) local GPFS and MPI I/O and (2) data transfers from the GPFS disks across the gigabit Ethernet switch and into HPSS.  
•	Our objectives were to demonstrate reliable, high performance I/O on White and from GPFS to local HPSS archive
Koelbel – Is all the I/O going in one direction?
•	It is all going to HPSS but there are reads and writes going on in GPFS.  In the context of the milepost we did not measure reads from GPFS to the White machine.  Sandia or LANL will pull data from White, across the WAN to a remote HPSS
Zosel - For local usage, the apps write much more than they read.  So we are looking at close to write-only data and that is driven by the ASCI program compute requirements.  Mark Gary will provide statistics.
Lucas – Is that 15 robots running stripes?  Answer: No, we only went to cache.
•	We think we are going to have to go much faster on the future platforms than we are going now.  The data rates we get depend on many factors.  We think we are talking about factors of up to 10-20% worse performance if I/O is done very badly.  Users will sometimes tend to do what is easy to get around the I/O problem and the aggregate I/O rates will differ.  We would like to develop a guide for I/O in the future for our users.
Johnson – Will these guidelines apply to the Q machine?  
•	I defer to the LANL folks.  In general, I think approaches to accomplishing good I/O on White would also apply to Q.
Koelbel – You are giving advice to users.  How strong is the correlation between your benchmarks and behaviors of the codes?  Does the additional compute time affect the I/O?
•	I would like to defer my answer until the next talks.  If we are trying to break GPFS before the codes do, I think the stress data will be helpful.  Breaking GPFS does sometimes mimic the applications

Tyce McLarty, Scalable Input/Output File Systems and Libraries
•	Our functionality objective is to enable the applications to do I/O as needed.  We performed stress tests to ensure that GPFS and I/O libraries are reliable.  Our demo was to show the best performance possible from scaled-up, MPI-IO and HDF5
•	PSE serves as the principle focal point between code teams and IBM I/O products and developers.  Our testing puts a tremendous load on the switch so improvements to GPFS result in overall improvements in machine stability.
Carlson – What do you mean when you say break?
•	Sometimes we mean kill one of the I/O servers.  Sometimes we hang GPFS.  To my knowledge, we have never actually lost data.
Louis – We bring the system down and then file a TMR with IBM.  We find a mixture of hardware and software issues.  A problem on the machine often becomes visible through GPFS (even though GPFS may not be broke). 
Browne – GPFS (General Parallel File System) is the canary in the coalmine.
Hedges – We see maybe 50-60% problems in GPFS and 20% problems in the switch, 20% in MPI-IO
•	IBM controls the software and maintenance.  Full scale testing of system software is done at LLNL.
Carlson – What is the bottleneck in the system?  Answer: Getting data through the switch.  The disks are on the I/O nodes.
•	Our major achievement has been improved stability.   All users benefit.  We found low overhead with GPFS
•	The total I/O rate of the system decreases by 65% (this is the bad news) from 4 to 16 clients.  We don’t know why this happens – it is our next target to work with IBM.  If IBM can make one process per node (for 256 nodes), they should be able to make it work for multiple processes per node.

Mark Gary, HPSS and the PSE Milepost
•	Now we are talking about getting data off the machine to the archive.  Our objective was to demonstrate White-to-archive data transfer functionality and capabilities at scale.
•	HPSS is a collaboration (beginning in 1993), a PSE project and a product.  HPSS started as a CRADA and five years ago became a PSE-funded and driven project.  HPSS is a scalable, parallel, next-generation storage system that is an IBM service offering and runs in 21 HPC centers around the world.  ASCI application requirements drive the HPSS project.
•	We performed five multi-hour demonstrations on 3 ASCI platforms and demonstrated functionality, reliability, scalability and high performance
•	File size had a major impact on our demonstration results
Johnston –Is there a special relationship between HPSS and GPFS?  Answer: No, not at this time.
•	If we want to go faster with HPSS, we build more concurrency.  Jumbo GigE jumped us up.
•	We demonstrated a scalable, high-performance archive for safe storage of the fruits of ASCI computational investment.
•	We have a sizable disk cache and we migrate it to tape on the back end.  HPSS has the ability to stripe directly to tape but we are able to bleed the disk cache fast enough (speed depends on the number of tapes in play – each tape sees about 13 MB/s)
Browne – Will this system keep up with a 100T machine?
•	We hope so.  We built HPSS to be scalable from the start.  There are numerous ways we need to be scalable (number of names, users, etc).
Browne – How often do people read things out of the archive?
•	We have about a 9:1 read:write ratio (13:1 last year)
Browne – Is the stuff that was read the same stuff that was written last (is it LIFO)?
•	That is very hard to answer.  My tendency is to say yes but we still have people going after 1960s data.  The disk cache is multiple RAID arrays using multiple technologies
•	Demonstrations verified the impact of file size on performance (large files are good).  HPSS developers continue to enhance system functionality, reliability, scalability, security and support of state-of-the-art technologies
Carlson – What is the back up strategy for a system like this?  Do you make time-dated or duplicate copies of files?
•	We provide a reservoir for people to send data to us.  We make duplicate copies of certain types of files (depending on the class of services).  There is no automatic move from GPFS to HPSS.  We are working now to allow users to bundle files before they send them to us.
Giles – As things are scaling up, do you see the users thinking about storage differently?
•	I have been in storage for 17 years and I find it amazing how little users’ attitudes about storage have changed. They are dumping more files and hanging on to them longer.  The size of the pipe we provide to the platform governs what they do.
Louis – The size of the disk on the 100T machine will be 2-3 petabytes, which is larger than the current architecture.
Brown – Bob Weaver tells us that his threshold of pain is over night archival.  As long as it takes no longer than that, he says he can live with it.
Johnston – You said that you had multiple classes of service on HPSS?
•	A user can set a class of service (if he knows what they are).  The idea is for us to make decisions based on size.  Users could make life difficult for us.  The main criterion we use now is size of the individual file.  Small files get dual copied (because there are so many of them on a single disk).

Ray Berg, DCE/DFS 
•	Our objective was to demonstrate SIERRA code build on White suing DCE/DFS to access the DFS code repository at Sandia
•	Sierra was selected because it is critical to Sandia’s weapon activity.  It is one of the Codes for the Complex and is a framework that allows coupling of diverse FEM codes for multi-physics simulations
•	Inter-site login and DCE/DFS are supported on White.  SIERRA is able to use DCE/DFS for local and remote data access, storage and management.
•	There are still issues to be resolved: Kerberos ticket forwarding and IP aliases, credentials management for batch jobs and regression testing, vendors need to coordinate DCE/DFS versions with the operating systems and it is not yet available for  some systems (e.g. Linux)
Johnston – What do you do about refreshing tickets for long batch jobs?
•	We haven’t solved that yet.
Lucas – Is SIERRA representative of other ASCI codes?
•	It is the most difficult framework that Sandia has
Browne – The framework is a set of components, underlying layers and physics layers on the top with a mind boggling linking problem
Zosel – SIERRA wants to have a single code repository that can be used to build from multiple platforms. This is different from having the source on the White machine.  
•	Operationally it is combining diverse models into a single model.  The details of SIERRA are not PSE but belong to the application group.
Merkle – Where does PSE stop and DisCom2 start?
Zosel – DisCom2 takes over with distance and high bandwidth.
Hale – The fact that to build SIERRA uses the development environment on White means that PSE is involved.  The DCE/DFS has historically been a PSE project.  DisCom2 has been in place for only three years.
Koelbel – Can I say that the reason we are talking about DCE/DFS and building SIERRA today (instead of tomorrow) is historical accident?
Hale – We will talk about this tomorrow as well.
Lucas – If SIERRA had belonged to LLNL, would PSE still have selected it for this milepost?
•	It is a good example of a whole collection of issues wrapped up into one.
Hale – The SIERRA environment is somewhat different from other codes in that they are trying to maintain a production environment on all of the platforms as a component of Codes for the Complex.  DCE/DFS is critical to that objective.

Mary Zosel, Summary Comments
•	We have provided you with copies of the Review Panel charter letter and definitions of the milepost
•	When we do this again for the 30T, how can our milepost definition be better?
•	We are asking for general recommendations for PSE
Carlson – You describe this as the initial software environment for the 10T.  Will there also be an initial software environment for the 30T?
Brown – Yes, that is what the users want.  We would also like to have mileposts for final environments.
•	We are working now on starting a decent definition for the 2005 milestones.  Part of the goal will be to show a consistent environment regardless of what platform you are on.  We will work the transition path needed to make these tools available in a consistent way to all users.  We welcome your suggestions on how to do this.
•	We provided some background information on page 2 of the milepost report.  We are heavily apps driven.
•	I believe we will hear from Dona and the others in response to your earlier questions tomorrow morning.

Wednesday, April 11, 2001		DisCom2 PRESENTATIONS 

Dona Crawford, Responses to Tuesday Questions from the Panel
•	The Panel presented the following questions to the PSE Leads with a request for additional information. I would like to answer those questions for you now.

1.	What is the context of this set of PSE mileposts (in the space of codes that solve the overall problem)?
•	The ASCI goal is 3D, full physics, full system simulation (primary, secondary, hostile environments) on a…
•	100T Machine by 2004 (1T, 3T, 10T, 30T, 100T)
•	Unfortunately, I was not able to find the list from the apps people identifying what they needed to meet these milestones
•	We are managed and organized by program areas with leads at HQ and at the labs.  Working as a TriLab management team was new to us.  PSE’s responsibility is to make the machines work.  TriLab management is now how we manage all ASCI work but it still poses challenges for us at times.
Johnston – Did the code designers identify the same things that are top priorities on your list of ASCI program components?  Answer: Yes and that is how we determined what we should work on.
•	We found that what the universities and others in the HPC world were doing was not going to work for ASCI
•	PSE focuses on the software to make the platforms work locally, DisCom2 is the process of extending access to the platforms, VIEWS is designed to emphasize how to make the data useful to the users, PathForward is designed to provide incentives to vendors to develop the enabling technologies we need to meet our ASCI goals
•	What software components are NOT on the list of things we are working?  Databases (because they are not TriLab in nature), portable global parallel file system, portable operating systems are some examples
Johnson – What is the relationship between longer term research needs and PSE?  Presumably everything in 2004 will not be perfect.  How do you coordinate and relate research needs with PSE?
•	In 1996, 2004 seemed a long way off but now it seems close.  We agreed last fall to plan to 2010.  We believe we can see our way to the 100T but what about beyond that?  We are more focused on deployment of applications than we are on research
Christensen – I should point out that there is a great deal of research focused on algorithms
Browne – What is the coordinating structure for ensuring that all of the ASCI program components work together?
•	We have a set of curves and barriers that provide a focal point for the ASCI program components to work together.  We determine whether we will seek to solve these needs ourselves or work with others in industry and academe to leverage their talents and resources.  We have an annual PI meeting.  We hold telecons routinely among the labs and have quarterly meetings.  In short, ASCI is a well managed, “well-meetinged” program.
Browne – Are there any joint milestones among the ASCI program elements?
•	Not specifically, but if you look at the Sandia mileposts, you will see requirements from VIEWS, DisCom2 so there are requirements from other program components embedded in the milestones for any given program component
Browne – Is there a single document for PSE that defines these needs?
•	Not in a succinct form

2.	What is the life cycle of these tools?  Will they be portable to Compaq?  Will they be portable to Linux?
•	All of the lab developed tools will  be released and  ported to all future ASCI platforms (yes)
•	PSE/PathForward tools are part of the market strategy for PathForward partners and will be commercially available for the foreseeable future (yes)
•	Other ISV tools, currently available on limited HW/SW configurations, will be worked on a case-by-case basis (maybe)
•	IBM tools will be available on IBM hardware (maybe)

3.	Who owns the SW development environment?  Who owns the actual development tools?  (frameworks as an example)
•	PSE owns the responsibility to acquire and integrate tools into the software development environment.  At each lab, the local PSE team is responsible to ensure the environment works on their platform(s).
•	Individual tools are owned by the tool developers 
Koelbel – What about tools moving backward to the other existing ASCI platforms?
•	If the apps folks want that to happen, we will make it happen
Zosel – We have some situations where IBM has refused to put features into existing systems (watchpoint in TotalView for example) so porting to existing systems is not possible.  We handle this on a case by case basis.

4.	Do these tools span the space of needed capabilities?  Will they be adequate for other ASCI codes?
5.	Are the objectives of these mileposts relevant to the future SW development problems of ASCI?
•	(These questions are answered together)
•	Codes were chosen from each lab to be representative of the methods, physics, and user communities required for ASCI application milestones and DSW
•	The codes have been and/or will be used in ASCI app milestones and are regularly used for stockpile simulation
•	The codes chosen are all unclassified
•	The users and ASCI program office selected the codes
•	The codes span 85-90% of the ASCI code work (all of the code work for the 2004 timeframe is now underway)

Other Comments:
•	Milestones are not intended to be comprehensive assessments of all aspects of a given component of ASCI.  Each one addresses a specific step, limited in scope, that has been judged to be a significant marker of progress in the overall program
•	PSE Metrics (functionality demonstration at scale):
•	Can codes from all 3 labs run on these new systems?
•	Can we demonstrate use of ASCI White using local, distributed and archival file systems from 1500 processors?
•	Can we demonstrate the availability and usability of tools from 1500 processors?
•	DisCom2 Metrics
•	Extend classified, initial code development environment for remote users in the TriLab
•	Easy remote access to information about the White environment
•	Basic data movement between sites at reasonable speeds
•	Remote job submissions and monitoring
•	TriLab coordinated operational support
Koelbel – How often do you revisit user requirements?
•	Informally, we do this daily (PSE and computer scientists sit on the code teams).  Formally we have meetings periodically.
Zosel – PSE has twice now gone as a TriLab group and met with every TriLab major development team and asked them about their needs, use of language, preferences for tools, etc.  Most recently we did this last fall
Browne – Do you have this assembled in a formal document?
•	Not in a formal document.
Brown – Each lab prioritized what we thought we heard and fed it back to the users for their review and comment.
Zosel – For example, last fall I asked if the users were beginning to use Java.  At what point will that become a formal requirement?
Crawford – At Sandia we review with the users on an annual basis.  We also revisit the milestones on an annual basis.  Things have not changed substantially since 1996 but we have made adjustments (the hostile environment milestone, for example).
Nowak – The 100T is actually the entry point (not the end point) for weapon system simulation.  When we talked with the design community about the problems we knew how to solve, we arrived at the 100T (sufficient memory to do what is needed).  Once we have that, the goal will be ubiquitous 100T computing and we will go from there.
Giles – This is very helpful.  Please explain the relation of the milepost activities to the overall computation activity.  Is this all about everyone chasing mileposts?  Are the mileposts the activity or just indicators?
Crawford – They are indicators.  SSP is the real work.
Thompson – PSE activities are not done in a vacuum.  They are coordinated with the users, apps and among the three labs.  This takes time to organize.
Reed – I was asked about budgets last night.  The current FY budget for PSE activities across all three labs is $51.7M (spread fairly equally, I believe).  We spend $46.5 M on DisCom2 and that goes principally to Sandia ($3-4M to each lab for people and then hardware is separate).  Collectively these two components represent about $100M out of the total $700M ASCI budget. The ASCI program components represent 1/3 of the budget, the computer centers another 1/3 and the apps are the final 1/3.   The ASCI Alliance program is roughly $25M a year.
Catlett – Yesterday we heard presentations from different toolmakers.  What types of interaction are in place to assure that the result is a targeted suite of tools?
Zosel – That is one of the new focuses in the ASCI PathForward Tool Initiative.  We also have the MPICH folks at ANL in the discussion.  The talk focuses on things that are needed for the tools to link together.  We have not yet brought in our platform partners but the coordination is well underway with the ISV.
Browne – How much funding does VIEWS receive?
Reed – The VIEWS budget totals $70M this year.  PathForward is $32M (in FY01)
Van Rosendale – The VIEWS budget also contains $10M for a visualization server at LLNL this year.

Art Hale, DC-1.1 Milepost Overview and Background
•	“Initial simulation development environment extended to the 10T system” = DC-1.1
•	DisCom2 provides a bride between designers/analysts and high-end distributed computational resources
•	Our strategies are to develop, deliver and deploy an effective and efficient production distance computing solution and to integrate computing capabilities distributed throughout the complex
•	The milepost we are discussing today is an initial step to integrating ASCI computing resources.  There are additional DisCom2 mileposts in FY02, FY03 and FY05
•	Historically distance computing has been painful : inconsistent access, lengthy file transfer times, limited interoperability of tools, different systems and uncertainty of support have all been issues
Carlson – Does DisCom2 incorporate policy aspects (such as who gets to run where)?
•	There is a mechanism for establishing policies and allocation of the machine at the ASCI Execs level.  I participate on the systems architecture group.  We have been concerned about the need to address these issues.
•	The milepost requirements were derived from the ASCI White Usage model.  We determined the capabilities needed to meet these requirements (some capabilities exist, others represent new features) and from those documents we develop an implementation plan.
Carlson – Are you trying to support collaboration among teams at disparate places or collaboration among local teams?
•	More the latter than the former.  Most of the apps teams are located at a single site (except for Sandia with Albuquerque and Livermore locations)
•	The formal testing for this milepost was completed on March 28th.  All milepost requirements were achieved.
Johnston – A year ago one marginal issue was the highly tuned (and peculiar) nature of the batch scheduling strategies.  Is this still an issue?
•	Each site continues to have its own scheduling strategy.  It is hard to talk about replacing any of those services now when they have taken so long for each site to deploy them and the grid services are brand new (we have zero experience with grid services in the user community).  De-emphasizing the use of the local scheduling mechanisms will have to take place over time.
Koelbel – How much coordination are you doing with other large distributed projects such as NSF PACI?
•	We have several efforts in place now (participating in the Grid Forum, for example, and using Globus). 
•	Major DisCom2 milepost accomplishments:
•	Awarded contracts for high speed (2.5Gb/s) WAN connectivity
•	Resolved initial parallel data transfer issues in TriLab WAN
•	Coordinated TriLab SSS-FTP deployment
•	ASCI GRID services (DRM and Globus) secured and deployed
•	Worked with Computer centers to provide coordinated support

Tom Pratt, DC 1.1 Milepost Review – SNL User Drivers and Distance Challenges
•	Our users are interested in memory, system reliability, performance, system stability, and interoperability with local resources and system support.  Each lab has in excess of $100M of local resources that can be leveraged with the remote resources.  Our users first want to know how to get started and then want answers when they don’t get the performance they expect.
•	We also have institutional drivers – each lab has a piece of this puzzle to solve.  Resource management, system security, risk management, agility to adjust to shifting priorities and define a path to the future all serve as institutional drivers for DisCom2.
•	Where we were: OC3 17MB/sec physical connectivity, Data transfer 0-2MB/sec, DFS 3+kbytes were blocked, no policy/procedure to get large amounts of resources, local accounts for remote users, three local authentication methods, three methods of handling remote accounts (still true), two models for account creation, remote web pages access required local accounts, two user support models (LANL, LLNL), no procedure to communicate between support organizations, two DOE offices for security approvals, three world class computing platforms with three different architectures (different back planes for messaging, for example), operating systems and queuing systems.
•	Robustness is the key issue.  Kerberos/DCE authentication got us a global name space.  DCE/DFS got us a successful file system from all of the labs.  The high speed WAN will give us performance and will allow for high speed, secure communication among the sites.  The grid is coming and – when things go wrong – there is now coordinated user support.

Steve Turpin, LANL User Drivers and Distance Challenges
•	The drivers that Tom described are fairly common across the complex.
•	We started with three sites with roughly equivalent computing capability (all in the classified environment).  Encryption is a major issue.  ESNet provided an OC-3 line and we put encryption devices at each end.  Each site had HPSS (configured differently)..  LANL had disk cache (unless the file was too large in which case it went directly to tape), LLNL had disk cache and Sandia had tape cache.  Those interfaces are encoded into scripts users need to do their jobs.  We had security filters in the original network that substantially impacted performance.  We also had filters on the routers that provided speed bumps (due to security).  Originally the OC-3 was adequate.  There was little motivation to do remote computing since the platforms were essentially equivalent.
•	I would like to describe a success story for DisCom2 that was Pre-milepost.
•	LANL completed its secondary burn code milepost eight months early.  Part of the reason was that LANL was able to run part of the calculation on Blue Mountain and part on Red (the calculations ran better on these machines).  5.3 TB of data was created and required 138 hours to move from Sandia back to LANL.  HPSS has no export ability so it was not possible to move this to tape (which would have been much faster).
•	Since we completed the milepost, we can now do end-to-end transfer at up to 100MB/sec.  We have authentication.  The network from California to New Mexico is in place.  The ends of the fibers are in place between Albuquerque and Los Alamos (they have not been tested).
•	We did an RFQ and Abba won the contract. But Abba has to build new infrastructure and Qwest has a loop in place.  Qwest had used all of the fibers in their northern loop so they brought in equipment to do dense wave division multiplexing (first time in this area).  They have deployed it in the northern loop and have given us four fibers (dedicated).  That is why it took so long to get the Albuquerque to Los Alamos connection.
•	To run the problem at Sandia, the code had to be ported to Sandia from LANL.  This meant that the LANL code team had to learn the environment at Sandia.  We took Globus, ripped out PDI, put in Kerberos as part of out Grid Services so similar transfers will now be transparent to the users using DRM.  A week ago a member of this team who is trying to run on White called me with an 85 GB file he was trying to get to LLNL across the OC-3 (WAN is not there yet).  We will go to OC-48 between Los Alamos and Sandia.  The users have given us a requirement of 55MB/sec to complete this milepost.  If there is sufficient funding, in the next fiscal year we will go to 10MB/sec.

Dave Wiltzius, White Usage Model
•	The objective of the usage model is to provide sufficient detail (but still be a concise document) to guide implementation across the three labs.  The goal is a common vision.  The second objective is to make certain that we meet user requirements.
•	The DisCom2 WAN came into operation this past February.  ASCI White went into limited availability a few months ago. We do not have a great deal of experience yet.
•	Beginning a couple of years ago we surveyed the users and fixed the user requirements last  summer
Johnston – Your budget is close to  $48M – which is several times the NASA and other budgets.
•	There are other components of distance computing that are included in that total.
Hale – In the low $20sM of that budget (roughly half) is dedicated to people and hardware for the WAN, DRM development.  The remaining half goes to cluster procurement for Cplant.
Johnston – At some point it would be useful to understand what the level of software and hardware involvement has been.
Lucas – It would be helpful to clarify that this is not Cplant.  We would like a first order breakdown (assume $300K/FTE) on the costs of meeting the milepost.  ACTION ITEM
Hale – I will have that for you tomorrow.
Koelbel – I understand the constraints and I sympathize.  What are the plans for updating the usage model going forward?
•	The usage model for the White will continue.   We will also begin to develop a usage model for the Q.  We have early drafts of usage models for Cplant, also.
•	Distributed computing is a milepost for us in 2003 and includes the entire nuclear weapons complex (meaning the plants as well as the labs).  Distance computing focuses solely on the three labs.
•	The requirements that frame the White Usage Model came from the extended use community.
Fox – Will you debug locally and then move the production code?
•	Yes, we recognize that there are significant local resources and that users will use them first
Zosel  - TotalView is available on all of the platforms.  The expectation is that people will do code preparation and debug on their local platforms. We will be standing up a smaller system (Ice at LLNL) to demonstrate the code and be able to make the code run on the IBM White.
Christensen – Most people run on the desktop to develop the code.
Koelbel – Assume we develop locally and are now ready to move the run to IBM.  If a bug pops up on Ice, how to resolve it?
Brown – We had an experience where a significant bug only showed up with AIX so we had to resolve it at LLNL.
Browne – The users will have a certain set of tools on their desktops.  Then they will go to the local production machine (which may have similar tools or different ones).  Then they go to Ice and then to White (Ice and White may have the same tools).  Conceivably the user will need to be able to use three separate sets of tools.
Zosel – We are migrating towards similar tools at each location (if not the same tools).
•	We believe that data movement will occur expeditiously on SecureNet except for the very large data sets that will be developed on the White platform.  SecureNet will not be loaded with large files coming off of White.  
Underwood – The test cases used to achieve the mileposts were derived from the key points on the White Usage Model.
Merkle – Can you describe the process for these points in the usage model?  Did you show the users lists of tasks?
•	Weapons designers are very busy individuals.  Each lab understood its own users.  We shared the document with the users.
Hale –Running the apps here refers to running the calculation to get results.
Lucas – We would like to see an interactive running of TotalView as part of the DisCom2 demo.  **** ACTION ITEM
•	White Usage Model Key Components:
•	Gather information
•	Data migration
•	Running applications
•	TriLab coordinated operational support

Jim Ang, Capability Release
•	Our objective is to define a process through which ASCI-developed and procured technologies could be integrated with existing capabilities into the user production computing environment
•	A key driver was developing a systematic method for assuring that the technologies were synchronized, integrated and deployed.  There needs to be a way to manage the technology insertion process.  
•	We look to the implementation plans for the technology development projects and then assess features that need to be released and deployed.  Curves and barriers address the integration of technology development; we believe that capability releases address the integration of technology deployment
Carlson – Is this usage model supposed to capture the user community requirements? This appears to be more of a computer center document telling users how to use the machine.
•	It is an iterative process. Some user requirements are not met (some users ask for more than we can provide) so there is give and take in the usage model.  The document gives us something in black and white that we can design to.
Hale – We cannot put words in the mouth of the users.  We tried to capture what the users tell us they need to do their jobs and combine it with the technology we know we can provide. The intent is to present a user perspective.
Koelbel – Have the users agreed to this document?
Hale – The content is based on surveys of the user community.  We have circulated it back to them and have heard some comments about things we do not believe we can provide at this time (remote visualization, for example).  We use the document for planning and also as a communication mechanism.
Wiltzius – Users have provided feedback but have made no major changes.
•	The usage model will always have a date on it and will change over time.  We will meet with the VIEWS community next month to discuss remote visualization
•	The role of the capability release is to facilitate and coordinate discussions about issue resolution associated with TriLab technology integration and deployment
•	Users have input to the capability release process through the use case analysis
Carlson – Who negotiates the CR process?  
•	Martha Ernest and I try to facilitate.  The program components own the releases.  The computer centers and the technology developers most frequently participate in our weekly conference calls.
•	Accomplishments:
•	The process works (it is flexible and will evolve and improve, key is good communication among the labs, balancing ASCI focus on R&D, integration and deployment)
•	Helped to establish the TriLab configuration management board
Johnston – At the review two years ago you presented some detailed use case studies.  Did that feed into this process?  Answer: Yes.  That is an ongoing thing.
•	The process of raising issues is not always comfortable.  The CR process is viewed as DisCom2-centric but its impact could extend to the deployment of non-distance computing technologies
•	Next steps:
•	Formalize user acceptance tests
•	Establish application service center (CR Stat)
•	Future targeted technology deployments:
•	Distance vis capability
•	Data services functionality
•	Distance computing support for Q
•	Extension of distance computing environment to plants

John Naegle, Parallel Network and Data Movement
•	Our vision for the milepost network environment:
•	Large data flows from White to Sandia and LANL
•	Order of magnitude user perceived improvement in WAN
Lucas – Last month at the Blue Light review, there was discussion about the relative growth of flops and bytes versus core storage.  Should bandwidth be growing faster, slower?
•	When we had equal compute platforms we did not need the network.  Now we may need relatively more bandwidth than we needed in our previous environment.  Our intent is to get this network (an order of magnitude improvement) in place and watch it carefully.  There is no true science to this – it represents a guess that includes what we thought the users want, what our budget could afford and what technology is available.
Catlett – Do you have an option where someone would compute at one site and then store the data immediately at a second site?
•	There is a possibility for that.  Right now we are assuming we don’t have that kind of demand.  Our immediate vision is to move to local storage or post process the data.  The functionality is there but we have to wait to see what the users want.
Johnston – if there is not much collaboration among the design teams, there is little requirement for remote visualization?
•	That question has come up again recently.  We need to discuss this and explore how we can use the parallel network for vis data (and not just basic files)
•	More tightly integrated network and user data movement tools to simplify network use
•	Continue to implement classified security capability while maintaining performance in WAN (Type I NSA encryptors are a requirement for this network)
Carlson – Does system security include authorized users within the community and protecting the data on the wire?  
•	Yes.  Most of the original speed bumps were due to the original architecture (including things like application gateways and hardware that could not perform the access control list (and work with the complexities of the lists)).  In one sense we have a different environment now. 
•	End-to-End Parallelism for Aggregate Performance
•	End-to-end is from disk/tape to disk/tape
•	No commodity encryption, disk and NICs sufficient to meet performance without parallelism
•	We knew our end user requirements for performance would be more than 622MB/sec at the present time so we knew we had to build an end-to-end mechanism for moving in parallel (so there would be no single point bottlenecks). We had to get apps that were capable of using these parallel pipes.  At some level the app must know the parallelism in the network.  Right now we have dedicated pipes.
•	Key features of parallel architecture:
•	4 way stripe, each limited by OC-12 encryptor to give aggregate of OC-48
•	Multiple GigE or ATM interfaces on hosts
•	Unique IP addresses on each path
•	Static ATM and IP routes to guarantee separation of data flows (no commodity mechanism exists to allow dynamic path separation guarantees)
•	Examples: ASCI Red (four way ATM network).  We have independent firewall boxes outside the external network in our unclassified system.
•	We are starting with basic file transfers (multiple processes to move parts of the file in parallel).  HPSS was the first apps we tried (example, from Red to HPSS in a parallel fashion).  There are also non-HPSS resources.  We ported the HPSS FTP function to non-HPSS platforms (SGI, DEC, Alpha, Sun, etc)
Catlett – HPSS breaks up the file system.  You mapped to the architecture of HPSS?
•	Yes, the only thing that HPSS did not do is provide a table of actual interfaces.  We leveraged all of the thinking that HPSS put into parallelism.  However, HPSS also brings much overhead in the form of inefficient data movement protocol
•	Three core scenarios for the milepost:
•	Interactive parallel HPSS FTP from White to Edison (Sandia vis and post processor)
•	Interactive parallel HPSS FTP from White to SNL HPSS Storage
•	Non-interactive “batch” client which is compatible with HPSS server for White to Edison (also works to HPSS storage)
Carlson – You modified SCP and BDS so you have that now?  What kind of improvement are you seeing?
•	Yes, we are seeing 20-30% improvement today over our current network.  We do not see that as sufficient to change at this time.  We saw just over 100MB/sec with the new and improved HPSS-based system.  We saw 120-130MB/sec with NetPerf (using lightweight protocol but SCP, BDS give the same performance).  BDS is similar to NFS but it is still proprietary (getting it to AIX is probably a nonstarter).  We have not done any of the security work needed to work among three sites.
Carlson –This is what I was discussing earlier when I talked about users providing the requirements.
•	This is a future possibility but we cannot implement it now in a secure fashion across the three labs.
Johnston – Are you using grid proxies for authentication?  Answer: No, we are using a Kerberos ticket.
Catlett – Presently, I have written estimates for connections between two place 500 miles apart for OC-92 for $1.92M/year.  
•	I understand.  We have focused on short-term contracts (3 years at OC-48) because the prices are dropping so fast.  It is highly dependent on where the vendor’s fiber is.
•	Bandwidth acquisition:
•	RFI went out early 2000 followed by an RFQ.  Four fully responsive bids came in (MCI, AT&T, Qwest, ABBA).  The lowest cost bidders ended up being the best value.  The contract was split into AT&T for LLNL to SNL/NM (delivered OC-48 on 2/9/01) and ABBA for LANL to SNL/NM (delivered OC-48 on 4/6/01).  The cost for the two contracts is around $3M per year.
•	End-to-end encryption research with NSA, DOD and GTE resulted in an ATM encryptor that we use now
Catlett – Are you tied into ATM for your WAN now?
•	We are tied into ATM for the encryptors but we can do other things in between.
•	Implementation challenges included network equipment, late bandwidth delivery, parallel data movement apps, encryption equipment, end host performance
•	Initially we tested the bandwidth with low level test sets, we also tested the network at the IP level (that was successful), we used a 237GB data set provided by a production user application, logical stripe width set to 16 to overcome latency.  We moved 237Gbytes in 2338 seconds = 101.8 MB/s.  All tests were performed during “lightly loaded” times (achieved 110MB/s during dedicated time).  LANL demonstrated similar functionality but without the bandwidth – expect to duplicate LLNL to SNL performance in next 2 weeks.
•	In the future we need to run things through the capability release process, implement end-to-end jumbo frame and improve robustness and flexibility of the network.
•	The basic components of the TriLab WAN environment are now in place.
Johnston – This is very good.  These plans were mushy in January of 1999 and you have solidified and implemented them since then.

Jean Shuler, Information and Support for Remote Users
•	Before DisCom2 each ASCI User Service Center supported local site users
Merkle – Do you think the users know whether they are talking to a DisCom2 or PSE person?
•	The users know they are calling the hotline.  We find whoever is needed.
Wiltzius – The DisCom2 budget contributes to Jean’s staff.
•	LLNL right now has 583 remote users (less than 100 secure users).  White is still in limited availability (25 users per lab is the maximum at present).
Louis – We have to run a synthetic workload yet on White (this is a contractual obligation that is part of the acceptance).  White will not be fully available until October.
•	Complete web pages provide information when customers are remote.  The manual describes expectations for users of ASCI White.  Web access provides a dynamic access status report (users have access 24 hours).  There are links to ASCI machines at the other labs.
•	Developed a TriLab user problem reporting process – the user can choose where to ask the problem.  No matter how the request originates, everyone who has a stake in the process is notified
Giles – Can the center people still talk with the users to answer questions?  Answer: Yes
•	We also have a trouble ticket system and we can search the data base by the user and responses given
Giles – How do you handle problems that straddle sites (moving massive amounts of data, for example)?
Pratt – That is handled by the sites affected.  If it is potentially a local Sandia issue, we look at it and determine how to respond.
•	A TriLab process has been established to provide access to the “most capable system.” 
•	Policy for use of White is established by the Policy Committee
•	Enhanced priority runs are handled through weekly classified video teleconferencing
Fox – Do you use WebX for remote diagnostics?  Answer: Not on the classified system
Browne – Do you have a hit counter on your pages?  Answer: Only on the unclassified side
Pratt – On the secure side it requires a tool that has not been approved yet.
Giles – Do you get direct user feedback on how you are doing (for the web-based stuff)?  
•	Only a feedback form on the first page.  We have seen an increase in use of the web pages.

Steven Humphreys, Grid Services/DRM Project
•	We do not want the user to have to worry about details that are not associated with the job that is to be done.  Local sites need to maintain control of their resources.
•	We have grid services deployed at all three labs.  At Sandia and LANL we have DRM application servers; at LLNL we have Globus services.  This allows Sandia and LANL to submit both local jobs and remotely to LLNL
Koelbel – There is no DRM server at LLNL because the other sites are trying to use ASCI White (rather than the other way around)?  Answer: yes
Carlson –Does the infrastructure decide where your program is going to run?   Answer: It can.  
Giles – When you also are looking at TB data sets – before the high-speed network – that means you keep them local.  With the high-speed network you have other major options.
•	The overall algorithm would want to build in cost considerations and other items to determine the best place for the run
Koelbel – What is DCPS?  Answer: LLNL’s resource management batch system
Browne – Each place has its own Globus name server?  Answer: LANL and Sandia have them.  
•	The computational grid provides common interface to computational resources.  Ideally, the user hooks a workstation to the grid and is immediately connected to the resources on the grid.  
•	The software architecture is a layered set of services.
Koelbel – Which parts of this diagram is DisCom2 contributing to?
•	Production wizard, monitoring, DRM services, PBS and Cplant and – to a lesser degree – Globus toolkit
•	Local resource managers (or batch queuing systems) differ from site to site.  This software provides the scheduling for batch jobs on each of the compute resources
•	Globus is a middleware toolkit that was originally developed by ANL, USC ISI and the Aerospace Corporation to provide basic grid services. It is a job submission protocol that is independent of the local resource manager. 
•	We modified Globus to interface with ASCI specific local resource managers, use NNSA approved Kerberos V5 for authentication between Globus companies.
Johnston – Do you use a separate set of queues for Globus jobs?
•	We use the same set of jobs as the standard queues.
Johnston – We had problem mapping to get a global ID.
•	We had to offer a user-to-user authentication capability. This was difficult for us, too.
Giles – I had the impression this morning that local users felt they owned certain blocks of time.
•	That was the old way of looking at things.  The sites each decide their own policies.  The ASCI Execs decide how to allocate resources and we use a “fair share” concept.
•	DRM is located above the Globus toolkit and provides workflow management, resource discovery and sharing and information service.  The DRM layer  is a set of Common Object Request Broker Architecture (CORBA) services that can be run in single server and distributed environments,
Johnson – Is GSF a part of Globus?  Answer: No, it is not.
Koelbel – We (the DOD Mod office, particularly) thank you for implementing Kerberos with Globus.
•	All of the changes we have made are being incorporated into the latest versions of Globus (PKI or Kerberos will be a choice).
•	The grid needs to know what resources are available at all times.  The GRAM updates resource state (resource configuration, system status, queue status) in the GIS.  The user no longer has to keep track of resource specifics or status.
•	The Production Wizard connects the user to the grid.  This process is based on XML (an XML specification goes out from the desktop to the server).  Right now we write the XML scripts for the users.  We use a GUI and one of the options is the name of the workflow file.
•	We have a basic set of grid services that are usable and we want to increase the user base.
Johnston –What are you going to do about workflow specification?
•	We have defined what needs to be in a workflow specification.  There is an industry practices group and we are using a simplified version of that.  
Johnston –Dagman and Condor are code languages that are presently used to specify workflow.  They are both worth looking at.
•	I will do so, thank you.
•	The broker provides a resource discovery for hardware and software resources that meet user specific requirements. The Broker can select the “best” resources based on user selectable algorithms
•	A user can submit a job with Globus but there will be no Broker.  A job submitted by the GRAM looks no different from a job submitted directly by the user.
•	Grid services provide flexibility.  Limited file staging is used primarily for input files.  The same mechanism that is used to kill a job could (conceivably) be used to suspend a job.
Koelbel – Do you plan to integrate computational steering and remote vis from here?
•	We have had a number of requests for restart files.  To make these things happen, we need better access to the vis servers.  
•	As we develop our user base and continue to work with the PSE folks we need to make sure that we are able to integrate TotalView and other tools into the grid.
•	Security services use GSF (Generalized Security Framework)  along with access control instruction and client authentication and authorization
•	Grid services have been successfully deployed at all three laboratories and are now usable.  The services are adaptable, programmable, secure and supported.
Fox – Are you interfacing TotalView and XML?  It would be helpful to develop more of the interfaces needed for a truly integrated system.
•	After the milepost, we have synchronized our development to the capability release process.  Detecting problems before they occur is important.  We want to enable people to work on long workflow packages without worrying about their user credentials expiring.  Beyond six months we want to grid enable non-compute resources (archival storage, data services and vis)
Browne – Assuming everything that you have told us about is real, this is cool.
•	It is real and we have the scars to prove it.
Johnston – Putting DRM on top of Globus is very nice
Koelbel – And, again, on behalf of others – let me thank you.

Martha Ernest, DisCom Test Case Results Summary
•	Review of five test cases run to meet the DisCom2 milepost (details in notebook)
Johnston – When we were moving files, we were told that one couldn’t reliably move a 10GB file across the Internet.  He found in the backbone traces that 1 in 50,000 TCP packets have undetectable errors in them and the TCP checksum is not long enough to find the errors.  The chances of finding such circumstances in DisCom2 among the labs is small but it is worth checking (particularly if you will be moving 250GB files).
Carlson – When were the goals of 36MB/s and 100MB/s established?
•	Our test plan was written last October and the numbers were set then.
Naegle – We thought we had a bigger margin because we were counting on the end-to-end jumbo frames.  After next month, we believe we will have much more leeway.
Hale – The numbers go back to user requirements to transfer data and the desire to have the same effectiveness for remote users as for local users.  I insisted that we pick a number that would be challenging and achievable.
Carlson – Is 36MB an SNL HPSS issue?
Naegle – Yes, we picked a number that is somewhat higher than what we can do locally and that pushed us into deployment of our disk cache system.  That will go up rapidly as we improve HPSS over the summer.
Johnston – Did the user forward Kerberos credentials before logging into White? Answer: yes
Koelbel – Did you look at all of the tickets, send in test messages or sample?
•	Tom Pratt looked through all of the tickets that went to Sandia.  
Pratt – I looked at every ticket.
Carlson – I was confused by the documentation where every line is crossed out.
•	We were going to cause a problem in Test Case V and instead decided to look at the data from January, February and March and see how well we had done.  
Carlson – Is it normal to have 10% of the tickets still open?
•	Essentially the ticket has been resolved but not closed.
Johnston – Why is there nothing in the milepost report about test case III?
Hale – It is in the summary report under running the applications.
•	The last three weeks in March we did daily status assessments to be certain that all tests were proceeding well.
•	In summary, all of the tests were passed.
Johnson – How close is the set of capabilities to routine?
•	All of these capabilities will be part of our May capability release

Art Hale, DC-1.1 Milepost Wrap-Up
•	We have put in place capabilities that constitute an initial distance computing environment and taken a step toward the TriLab integrated ASCI computing environment
•	Future work:
•	Feedback loop on how we are doing
•	Data migration, manipulation and storage
•	Running the application (richer job creation, submission, control and monitoring environment within  Production Wizard)
•	TriLab Coordinated operational support
Lucas – Are you planning to work on user generated restart?
•	Workflow right now is serial and does not generate the follow-on job.  We are not promising system generated checkpoints and restarts
•	We will provide the budget numbers you are asking for tomorrow morning

Thursday, April 12, 2001	ANSWERS TO PANEL QUESTIONS ABOUT PSE and DisCom2

Mary Zosel, PSE Budget Breakdown
•	PSE Numbers from April 2000 – April 2001 (for those actions associated with the White software environment)

Area	PSE FTEs	Leveraged to 30T	
Compile/Build	 .25   LLNL	KAI and OpenMP	
Profiling	 .1     LLNL		
Debugging	1.75 LLNL + .5  LANL	Etnus PathForward Project
$1.95M (PF budget)	
Memory Tools
•	ZeroFault
•	Great Circle
	 .2  LLNL	Strong interest in ZeroFault (unfunded requirement)	
Performance Tools
•	MpiP
•	MPX
•	Vampir
•	Umpire
•	Tool Gear	2.1 LLNL	PSE will port KAI/Pallas PathForward project	
RTS testing	  .85  LLNL	To be included inTriLab test  suite	
HPSS (includes HPSS – related activities such as RAIT)	7.5	LLNL + 2.5 SNL  + 6.5  + 3.0 ORNL
Includes all HPSS development	All leveraged	
Parallel IO/GPFS/MPI-IO	 3.75 LLNL	Test suites relevant to 30T	
DCE/DFS Environment for White and Sierra build	 0.5 LLNL + 0.5 SNL		

Notes:  Etnus, KAI/Pallas and MSTI SWPF projects all leveraged to the 30T
	 LLNL and SNL investment and interest in Compaq systems
	 FY01 PSE budget splits:
		LLNL   $11.917M
		LANL   $12.605M
		SNL     $11.705M (PSE) + 12 (enabling tech)
	Total FY01 PSE Budget = $49M
•	All of the three labs have significant Compaq systems so putting our tools on Compaq platforms is not really an issue
•	ZeroFault – I am told – is potentially interested in porting to other platforms and adding features if we provide some funding.  We continue to buy more licenses.
Koelbel – What about Purify?
•	Everyone loves Purify but the owners have stated no AIX.  They have priced themselves out of the market for porting and making their code parallel.  People like the Purify functionality.
•	This year’s project plan has porting MpiP, Vampir and the other performance tools to the Compaq as an objective
•	HPSS supports Compaq
•	Parallel I/O is presently IBM specific but could be leveraged in the future with additional work
•	The budget is roughly $12M at each lab.  About half of the PSE effort over the past year is reflected here – the other half is in solvers and other work
Merkle – The queuing system and configuration management?
•	There is an active effort supported out of production computing (50%) for the queuing system.  Configuration management tools are being used by all of the ASCI apps (the tools are strongly code team specific)
Ang – The configuration management repository is back at Sandia which is why certain tools were selected for the Sierra code build
Carlson – If I take the I/O projects out (HPSS, parallel I/O) – this appears to be a small effort in terms of FTEs.  Could you use more?
•	Right now this group is limited by finding talent. We have lost a couple of people and have a retirement coming up.  We do leverage with Livermore computing – it is not evident who is paid by PSE, LC or CASC

Art Hale, DisCom2 Budget
•	$46M at plan (reduced by 1.5 for S&S)
•	Distance computing solutions = $25M includes the following (Relevant to this Review)
•	$4M HW (networking) + 
•	$3.7M for HS-WAN (charges for the line and ESNet) and SecureNet + 
•	$2M for IP encryption + 
•	$2M for Y-12 and KCP + 
•	$4.5M for networking development and testing + 
•	$4.5M operational activities)
•	Integration of Computing Resources = $9.5M (Relevant to this Review)
•	$2.5M for CR process, integration and deployment activities) + 
•	$4.3M for grid services (development, test and deployment, $400K contract for Globus) +
•	$1M for miscellaneous security
•	Computational capacity = $17M
•	$10M HW
•	$5M SW development and contracts
•	$2M for Cplant integration and test (alpha boxes)
•	Program management = $1M
Koelbel – Is the PSE program management number similar?  Answer: Yes
Lucas – Does this pay for SecureNet for all of the labs?  Answer: Yes
Hale – At Sandia we do not use SecureNet for regular Internet service
Wiltzius – DOE pays a certain bandwidth for SecureNet and ASCI pays for the bump up.  If there were no ASCI needs, the capability would be a T3.  We are looking at bumping our service up to OC-12.

End of open sessions.

Notes taken by Tina Macaluso, SAIC on April 10-12, 2001
Distributed to ASCI Execs, PSE Execs and DisCom2 Execs on April 14, 2001