Overal the last month we have seen a dramatic change in the BBH Grand Challenge computer usage. Previously, we were running on the Pittsburgh's C90, San Diego's C90 and Cornell's SP2. Currently we have been developing, and running the Grand Challenge code, the ADM code, on NPACS 8 processor, 1.6 GB Mips 10000 SGI. Lately, the average load average for this machine has been well over 7. This code not only runs on NPACS SGI, but also on all other SGI's, as well as Crays. We currently run on this machine because we are using this as a dedicated machine for black hole development. This makes it far easier for us to use compared to the Cray C90 at the Pittsburgh supercomputing center. Since we can run interactively, we do not have to wait several days in order for a job to be started. Because of this increase in turn around time, we have been able to make several runs which show the the ADM code is stable for linear waves with reflective, and periodic boundary conditions.
The ADM code is written in Fortran 90, with over 28K lines. It has close to a hundred different modules to run such initial conditions as black holes, linear waves, strong field waves, etc. One of the problems with this code is it's memory usage. We are currently using over 112 3d arrays, which make it difficult to run accurate/stable runs on most computers. For example, to run a 129^3 simulation, we require over 1.9GB of memory, and a 65^3 simulation requires 246MB of memory. Since we are running parameter surveys to examine the stability of this code, we need to be able to run a very large amount of simulations. Therefore, we have implement our own form of parallelism, of where we let 7-8 jobs run at once on NPAC's SGI. Once a run is finished, we submit another jobs to run. Soon we will start using DAGH to run simulations faster, since it parallelises the code, and larger (since we will start running on SP2's, T3E's, etc. NPAC's SGI will still be used extensively for code development, and to run smaller simulations
This computer was built from scratch, (Marek can write this section).