HPF AMR implementation

• Using data parallel implementations, such as HPF, of the finite difference methods used in T2, allows us to achieve a very efficient implementation.

• The PDE solvers for the Einstein equations are computationally expensive, and this will hide costs of intergrid communications.

• Since HPF does not change the semantics of the code written in Fortran 90, there are easy to debug and optimize. HPF codes are also platform independent (when a comparable number of processors is used). Since HPF is a new technology, we expect the compilers will mature soon which allows us to capitalize on progress in both compiler and hardware technologies with significant effort to maintain and upgrade the codes.

HPF implementation strategy

• Data structures

– The grids on different levels of refinement are arranged as a tree structure. The tree is replicated on each processor. When vendors supply derived types and pointers to distributed arrays in HPF, we can implement this data structure. Currently we use a flat structure and grids are identified by their unique grid id.

– Each grid (a node in the tree) is a 3d mesh distributed (*,BLOCK,BLOCK) onto 2d arrangement of processors (PY by PZ), that is, each processor “owns” (nx,ny/PY,nz/PZ) section of each grid. Several grid functions are defined on the grid (34 in case of the T2 code).

– We expect that fine grids are of a comparable size as the base grid. We must deal with an optimization problem that involves regridding ratio, regridding frequency, and clustering efficiency, while preserving stability of codes, on one side and parallel efficiency on the other.

– ¿ operations on a single grid.

1. time step (hyperbolic solver)

2. multigrid elliptic solver

3. truncation error estimate followed by clustering

– Both hyperbolic and elliptic solvers can be efficiently implemented in a data parallel fashion (HPF). Truncation error estimate, thanks to the shadow hierarchy simply reuses the code developed for the hyperbolic solver. Clustering is more challenging (here the problem is the HPF global name space rather than data paralleism, therefore HPF extrinsic procedure will be employed).

– Operation on the hierarchy of grids.

1. Dynamic memory management: HPF inherits all Fortran 90 features which makes memory management easy (that resembles more C than Fortran 77). Unfortunatelly, pointers to distributed arrays are yet not implemented and we have to introduce some work-around methods at this time.

2. 2. Prolongation and restrictions: these are simple to code data parallel operations. However, since the source and target grids are typically not aligned it may induce quite expensive communication. We expect them to be small as compared to overall computational costs, though.