\documentclass[letterpaper,11pt]{article}
\topmargin=-.50in
\oddsidemargin=-.25in
\evensidemargin=-.25in
\textheight=9in
\textwidth=6.5in
\begin{document}

\begin{center}

\textbf{High Level Descriptor-driven Integration of Prototyping,
Communication, and Large Parallel Linear Solvers for Learning and for
Simulation of Physical Systems} \\

                        M. Choptuik\\
                        R. van de Geijn\\
                        Richard A. Matzner\\
               The University of Texas at Austin\\

                        Scott Klasky\\
               NorthEast Parallel Architecture Center\\
                      Syracuse University\\
 (subcontracted participation)\\

                       \textbf{Project Description}
\end{center}

\textbf{Introduction:}

The development of scientific and engineering computational models
involves discretization of the differential equtions, followed by 
evolution of the code and evaluation of results.

Each of these steps can involve very complicated procedures at the
lowest, technical level.  However, what the investigator wants to
control is typically at a much higher, abstract level.  For instance, repeated
experiments may be run using different discretization prescriptions.
The task of modifying such prescriptions at the level of manipulating
matrices or loops is very tedious and error prone, and can be very
profitably taken over by an automatic system driven by a high level
language, enabling the scientist to concentrate on the fundamental
features of the algorithm, rather than the details of the algorithm..
Similarly, study of graphical output is essentially the only way to
understand even one-dimensional situations and becomes essential in
multi-dimensional simulations.  Automatic, interactive and
collaborative graphical presentations thus can dramatically improve the
ease (and speed) with which computational methods are tried, analysed,
understood and, if necessary, modified. 

In the current computational context, parallel approaches are required
to achieve competitive performance.  Thus, we require support for
parallelization, for parallel I/O and for visualization of large
parallel data sets, and we require large, parallel, \textit{linear} solvers.

Precisely these same features are desirable in the teaching of
computational techniques to a new generation of scientists.  It
certainly is important that students understand discretization schemes
at the lowest level, and they can be taught. But it is important to
provide to the student rapid, reliable access to prototyping tools that
can describe the fundamental features of the desired code, without
explicit low level writing, can handle parallellization and adaption,
can efficiently run on todays parallel machines, and can be viewed
responsively, and typically in a collaborative setting.  The integrated
environment here will be demonstrated in computational science classes
taught at The University of Texas, and at Syracuse University, and will
include collaborative instruction, initially in a research environment,
eventually in a classroom setting, between the Center for Relativity,
The Department odf Computer Science at The Universityof Texas, and at
Syracuse University. .


The approach we propose here is to extend existing functional examples of
these tools, to parallel systems and to multi-dimensional problems.

The existing systems include:

                 RNPL (Rapid Prototyping Numerical Language), which
                 supports high level description of discretization
                 methods, and handles many ``housekeeping" features of
                 the coding, and can genetate full update code if
                 desired. \cite{RNPL}

                 DAGH   (Distributed Adaptive Grid Heirarchy), which
                 supports automatic parallelization, load balancing, and
                 adaptivity. \cite{6}

                 PLAPACK, a library supporting parallel solution of large
                 linear systems and algebraic eigenvalue problems.\cite{PLAPACK}


                 SCIVIS, which is a collaborative visulization and data
                 manipulation tool. \cite{SCIVIS} \cite{SCIVIS2}

These tools will be developed and extended to handle development
and analysis of large, parallel systems, and to provide responsive
tools for direct and collaborative teaching of computational science
concepts and techniques..



The acronym RNPL stands for \textit{Rapid Numerical Prototyping
Language}\cite{RNPL}. It is a language for expressing time-dependent systems of
partial differential equations and the information necessary for
solving them using finite-difference techniques. It has advantages over
traditional programming languages such as C and FORTRAN because it only
requires the user to enter the essential structure of the program while
it fills in the details \cite{RNPL}.

RNPL can produce complete working programs, or a ``skeleton'' that the
user can complete. The programs include facilities for reading
parameters from a file, interactive control over output times and
functions, memory management, and state dumping for calculation
interruption and restart.  The compilable code generated by RNPL is (at
the user's descretion) in C, C++, or FORTRAN.

Thus, RNPL provides the capability rapidly to generate C, C++, or
FORTRAN code from high level specifications.  The current implementation
of RNPL introduces a number N of grid functions,
$\phi^{A} [x^{\mu}]~~~~ A = 1 \cdots N;~~\mu = 1,~~\mu =1,2,~~$ or$~~\mu =
1,2,3.$ on a Grid, G, labelled by $i,j,k$  (or a superindex $I, I = 1...
$(number~of~grid~points)).  The state of the system is then described by
$ u^{\iota} \iota = 1 \cdots N \times $(number~of~grid~points).

Notice that we consider the temporal integration separately.  We discuss a
finite-difference evolution of the temporal dependences.  The finite
representation of the system (finite difference, finite element ...)
discretizes the system into a matrix acting on the state vector.

Existing RNPL implimentation solves this system in the following way:  The
basic set of equations:

$ f_{\iota}(u^{\kappa})  = 0$

is iterated:

$ u_{i+1}^{\kappa} = u_{i}^{\kappa} - 
f_{i}(u_{i}^{\kappa} )/ \left(\frac{\partial
f_{i}}{\partial   u_{i}^{\kappa}}\right. $

This iterative update succeeds in many contexts, but in some contexts
(particularly with strong inhomogenieties in the domain) may not converge.
The solution is to supply a general linear solver for the problem. 

Our approach is to use PLAPACK, which we will extend here to large
sparse parallel systems.


\textbf{PLAPACK}


Coding parallel algorithms is generally regarded as a formidable
task. To make this task manageable in the arena of linear algebra
algorithms, the Parallel Linear Algebra Package (PLAPACK) \cite{PLAPACK}
is an infrastructure for coding such algorithms at a high level of
abstraction. It is often believed that by raising the level of
abstraction in this fashion, performance is sacrificed.  Indeed there
is a performance penalty, but by coding at a higher level of
abstraction, more sophisticated algorithms can be implemented, which
allows high levels of performance to be regained. This has been shown
to be the case for the parallel solver package implemented using
PLAPACK, which includes Cholesky, LU, and QR factorization based
solvers for symmetric positive definite, general, and overdetermined
systems of equations, respectively.  Performance comparisons with
other such packages, in particular ScaLAPACK, shows there is no real
performance penalty when coding at this high level of abstraction.

In a dissertation by Tim Collins (Ph.D. in CS, UT-Austin, 1995,
co-supervised by J.C. Browne and R. van de Geijn), hierarchical
specification of matrices was shown to be a highly effective way of
storing and manipulating large sparse linear systems arising from
regular and irregular meshes \cite{Collins}.  In that research, type
information for submatrices in the hierarchy was also encoded and
exploited, allowing for highly elegant and compact implementation of
sparse iterative and direct linear solvers.  In particular,
type information allows a solver to take advantage of structure
in subproblems.  The hierarchical specification allows
for pruning of the matrix where it has zero entries.  In addition,
it allows for natural detection of "supernodes" in sparse
direct methods.

While PLAPACK was designed for \textit{dense} linear systems,
there are a number of reasons why PLAPACK is a natural candidate
for integration with RNPL and DAGH, and systematic implementation
of a hierarchical matrix infrastructure:

- The matrix distribution used by PLAPACK is \textit{physically based}
by which we mean that the distribution of the matrix is induced by the
distribution of the physical domain for which the linear system is to
be solved.

- Such domains are frequently partitioned hierarchically, e.g. in
nested disection methods.

- It is uniquely appropriate for implementation of direct methods for
solution of sparse linear systems since physically based matrix
distribution implicitly specifies the distribution of dense interface
matrices to distributed memory architectures.

- PLAPACK uses an object based programming paradigm, allowing
hierarchical specification of matrices to be encoded using an elegant
and simple interface.

- PLAPACK shares a common communication layer with DAGH, the 
Managed Message-Passing Infrastructured developed at UT-Austin
by Carter Edwards \cite{MMPI}.

- PLAPACK allows elegant and efficient implementation of the
orthogonalization of vectors that define Krylov subspaces as occur in
iterative solvers like conjugate gradient and GMRES.


\textbf{DAGH}



DAGH (which stands for Distributed Adaptive Grid Hierarchy) \cite{DAGH} was
developed as a computational toolkit for the Binary Black Hole NSF
Grand Challenge Project. It provides the framework to solve systems of
partial differential equations using adaptive finite difference
methods. The computations can be executed sequentially or in parallel
according to the specification of the user. DAGH also provides a
programming interface so that these computations can be performed by
traditional Fortran 77 and Fortran 90 or C and C++ kernels.

\textbf{Scivis}

Scivis is a client-server data visualization and analysis system, which enables
users to manipulate data from their scientific codes.\cite{SCIVIS2} One of
the major advantages of Scivis is that it is written in Java, so that it works
on all computer platforms.
Scivis uses a client-server model for data visualization. Users can
send the data to the server anywhere sockets are supported so that
the Scivis system can be used as a front end to other applications that
originally produce the data sets. The application producing a data set
can write the output data to a socket, which is connected to the Scivis
server. Scivis then allows the user to analyze and visualize the data.
The system is flexible and very responsive.  The system allows
vsualization rendering on remote fast servers, and only polygon lists
are transported for the interactive visualization of 3D objects.

One of the key Scivis features is called \textit{User-Definable-Filters (UDF)}.
Filters are
user definable actions,  performed on data set(s) to create new data
based on this action.  (For instance a filter may take a derivative in
space and/or time, of the original data, to create a  new dataset.)
The Scivis system contains Wizards to help enbale users 
design their own filters. Filters in Scivis can be created \textit{on-the-
fly} so that users can immediately apply these new filters to her/his dataset.
Since the Scivis system is
collaborative, it is possible to create a powerful platform where many
different users can can evaluate the performance of collaboratively
developed filters.

Another major feature of Scivis is that it is collaborative. Scivis currently
uses it's own collboration system, but will 
eventually switch from using its own
collaboration system, to one which takes advantage of
the TANGO system.\cite{TANGO} Using TANGO, Scivis will become part of
a very powerfull collaboration system which will allow it's users to take
advantage of many of the tools already introduced into TANGO. For example,
TANGO has many features such as shared web-browsers,
whiteboards, audio-visual communications, etc. TANGO has been incorporated with
Web Wisdom to allow for a new virtual University.\cite{WebWisdom}
This has allowed NPAC to offer courses at university while teaching them from
Syracuse University.

Since filters in Scivis are user-definable, Scivis users Serilization in
Java to tranfer the algorithms to other collaborators, in case the 
collaborator does not have the filter being used during the collaboration.
Using this collaborative framework, users can exchange
their data and their own personalized filters. Users of the system
collaboratively are identified as Masters, or Slaves. A Master can
modify data, a Slave can only view data. Multiple Masters are allowed in a
collaborative session.


\textbf{Proposal:}

Working collaboratively, and using the tools we are developing, 
Choptuik, van de Geijn, Matzner and Klasky will develop and extend the
four components of this effort into a unified coherent development and 
pedagogical tool for large scale scientific computational development
and for collaborative education and collaborative computational simulation.

\textit{\textbf{Extensions include:}}

RNPL extension to include automatic solution of the generated equations
by linearization and full solution of the  equivalent linear-system
(matrix) problem.
 
DAGH extension to inclusion of large linear system solvers such as
PLAPACK, and PLAPACK extensions.

PLAPACK extensions to parallel algorithms for the solution of sparse
systems of linear and nonlinear equations arising on  structured and
unstructured meshes, as well as for  gridless methods, through
extensions to PLAPACK that express sparse matrices as hierarchical
matrices.  This will include routines which recognize the sparsity
structure of the matrices, and adapt the algorithm accordingly.

SCIVIS extension to large parallel dataset manipulation, and to user
manipulation and visualization of extremely large parallel datasets.

\textit{\textbf{INTEGRATION:}}

The components described above will be integrated into a coherent
environment. RNPL will be extended to directly produce DAGH code, and
will be modified to interface the sparse-solver development of PLAPACK. 
DAGH will be extended to incorporate large parallel solvers through
PLAPACK. RNPL and SCIVIS will provide theuser interface element to the
developer.

\textit{\textbf{APPLICATIONS:}}

The environment as conceived will have broad applicability in the
development of simulation codes. The 1D description currently in RNPL
will be extended to full 2- and 3- dimensional support. Applications
include: One- and two- dimensional simulations of the behavior of
matter in strong gravitational fields, and in particular the appearance
of critical phenomena in gravitational collapse, and simulations of
3-dimensional strong-field gravitating systems.

   The topics described in this proposal address the KDI topics listed
   in the announcement:

   Knowledge Networking:
   -focus on attaining new levels of knowledge integration,
             information flow, and interactivity among people, organizations,
             and communities.  
   -enhance communication across disciplines, languages, and cultures 
   -improve the processing and integration of knowledge from
             different sources, domains, and non-text media types 
   -increase the effectiveness of teams, organizations,
             classrooms, or communities that work together across
             distances or over time 
   -deepen understanding of the ethical, legal, and social
             implications of new developments in connectivity 

   In particular the research would focus on interactive and collaborative
   tools for rapid development and analysis of multi-dimensional
   structured adaptive PDE codes.  This constitutes Knowledge Networking
   Foundational Research in:
   -Computational aspects of distributed intelligence: dynamic
             task allocation, interaction, coordination, process and
             organization representation, collective learning, consistency
             management, protocol, negotiation 
   -Managing heterogeneity and interoperability in dimensions
             such as syntax, semantics, scale, and structure 
   -Large-scale remote data acquisition, distributed data analysis,
             experiment and sensor control, and simulation; especially
             interactive and real-time aspects 
   -Distributed knowledge: sharable ontologies, processes for
             distributed classification and taxonomy, collaborative
             knowledge construction, representation and filtering tools,
             digital libraries and repositories across disciplines and
             application domains, and translation of representations

   In The New Computational Challenges, these topics contribute to: 
   -Interplay Between Computations and Data

\pagebreak

\noindent{Matthew Choptuik and Richard A. Matzner}

\noindent{{\bf Binary Black Holes: Coalescence and Gravitational
Radiation, ASC/PHY 9318152 (ARPA Supplemented)}}

\noindent{1. Summary of Previous Results} \\

The ASC/PHY 9318152 Grand Challenge Project was initiated in September
1993. Richard Matzner is the Principal Investigator and M. Choptuik is
a coInvestigator on this grant.  More than 100 publications have so far
resulted from the Binary Black Hole Grand Challenge Alliance.  (see:
http://www.npac.syr.edu/projects/bh/) Three of the technical components
to be extended here (RNPL, SCIVIS, and DAGH) were developed in the
context of the Binary Black Hole Grand Challenge Alliance.  Firther the
research of that Grand Challenge presented and presents a suite of
pedagogical and research applications for the integrated environment
proposed here.

The purpose of the Alliance is to compute the interaction of
black holes and the gravitational radiation produced. The Alliance
developed a parallel adaptive structure to take advantage of the fast,
large computers being promised. Computer science structures, such as
data structures definitions, and interface definitions, to direct the
construction of the ``toolbox" (which has become the ``problem solving
environment") were immediately addressed. One feature was adaptive mesh
refinement, which was immediately demonstrated in 1-d by
Choptuik$^{\cite{Ch94}}$, and developed within the
Alliance$^{\cite{MSW95, H95}}$.

Initial effort concentrated on High Performance Fortran, which is an
extension of FORTRAN90$^{\cite{8}}$.  The Alliance began with a
collection of legacy codes, and these codes were successfully
expressed in HPF (High Performance Computing)$^{\cite{8}}$.
This work was carried out by
computational scientists (especially Masso, Seidel , Cook, \ldots) and
computer scientists (especially Haupt, \ldots). Further, the
``toolkit'' effort produced early results$^{\cite{CFH94}}$.  However, we were
aware of the need to accommodate parallel computing, and immediately
undertook efforts to do so.$^{\cite{NC95}}$.  It became clear that the
limitations of HPF inhibited easy implementation of adaptive parallel
structures such as would become necessary in the Alliance codes.
Thus, an effort was begun to develop DAGH (Distributed Adaptive Grid
Hierarchy) which was written in C and built on an underlying structure
called SDDA (Symmetric Distributed Data Array).

In early 1996 the HPF efforts were concluded, and concentration
centered exclusively on DAGH development. DAGH consists of a C++
driver which handles parallelization (it uses space-filling curves to
convert from the conceptually 3-d computational space to a linear
layout of processors). It also handles adaptive refinement, given
routines to define an error measure and threshold$^{\cite{PB95, PB96,
PB96b,PB97,PB97b,PBEK97}}$.  It contains standard
clustering routines, as well as injection and restriction routines, to
communicate at the boundaries between different levels of refinement.
In most applications as intricate as computational relativity,
however, specially-tuned programs have to be supplied by the
computational scientist. DAGH is written so that this C++ structure
calls application science modules which can be written in C or in
FORTRAN$^{\cite{6}}$.  In the Alliance, the practice has been to write
these codes in FORTRAN90. Codes are typically written in a unigrid
context by the developer, following certain rules: no global
variables, array bounds passed in at call, etc., that guarantee
parallelizability, and with a FORTRAN90 driver which calls the
various science modules. We have written a script which reads the F90
driver and constructs a DAGH driver to run the program in parallel
(adaptive refinement is being debugged). This means that we have a
fast and efficient development process which can be translated
immediately into production-style DAGH code for large-scale
development or production runs. DAGH has been adopted into other
research applications$^{\cite{WYWADPS97,PWPWW97}}$ and has been
extended in other research directions$^{\cite{34}}$.  Relativity, i.e.
applications-science use of the computational resources, began with the
set of legacy codes referred to above. A period of testing and evaluation
ensued. A number of systems were studied: spherical systems, typically with
scalar radiation to provide a dynamical
field;$^{\cite{35, SST95, SST95B}}$
initial studies of axisymmetric vacuum collapse;$^{\cite{37}}$ and
collapse to black holes with matter sources$^{\cite{S+T92}}$.  
Fully 3-dimensional
data setting and initial computation was carried out by Matzner and Laguna
in cosmology$^{\cite{39}}$ using an ADM (g-dot, K-dot) code developed by Kurki
-Suonio, and a similar code was used at NCSA$^{\cite{P.Anninosetal}}$.%old41.
 Additionally, we had expertise on 2+1-d characteristic
codes, which allow integration all the way to radiative
infinity\cite{GPW94}.  Work soon began on a 3-dimensional
characteristic code. A number of technical approaches were studied to
expand the computational science resources available to the
Alliance$^{\cite{L96}}$.  Some of these took the form of internal
reports$^{\cite{BGPWX96, GPWS97, GPSWE97}}$.

Validation of codes proceeded in the 1-d case in work by Evans and
Coleman$^{\cite{E+C1994}}$, who studied gravitational collapse of a
radiation fluid, and found critical phenomena related to the
Choptuik$^{\cite{46}}$ effect. Development proceeded on broad paths
with a number of test and example cases used to develop technique for
the Einstein equations and for the important problem of radiation
extraction$^{\cite{AST94, AST95}}$.  Critical Phenomena remain a topic of
current interest, and code development, especially in the spherical case,
has continued to touch on this problem$^{\cite{Ch94B, Ch95, MC96, CCB96}}$.

There are a number of ways that data can be set for multiple black
hole configurations. Cook et al$^{\cite{52}}$ had, prior to the
initiation of the Alliance, shown computationally convergent methods
of preparing data, based on the ``classic'' approach of using the
York technique on conformally flat backgrounds. Cook extended this to
a study of quasi-circular black hole orbits$^{\cite{C94II}}$.

Early 2-dimensional black hole collision simulations were done using
maximal lapse (a singularity-avoiding coordinate
system)$^{\cite{ADMSS95}}$.  A number of these studies were carried out,
and laid the groundwork for the considerably more demanding 3-d
simulations: These included studies of distorted and rotating black
holes,$^{\cite{BS94a, BS94b, BS95,ES_H96}}$
 both vacuum and from collapse $^{\cite{AST94}}$, and a
thorough development of the dynamics of the collision$^{\cite{AHSSS93,
AHSSS94, AHSSS94T}}$.  These studies (based also on
previous work by Shapiro and Teukolsky$^{\cite{S+T92}}$) culminated in a
thorough description of the surface of merging black holes,
explicating many previously unclear aspects of the
process$^{\cite{MSSSSTW95}}$.  A fuller exposition subsequently was given
in$^{\cite{M95, ADBLMSSSW95, LMSSW96}}$.

An important aspect of computational physics is to compare with other
(perhaps approximate) methods$^{\cite{CSS1995,BCKP1997,KLPA97,KLP97B,KLP97}}$. 
Perturbation theory of distorted black
holes surely describes the late evolution of a black hole merger. A
number of studies have been carried out predicting radiation in
situations that can be treated by close (standard perturbative) or
other approximate methods. For axisymmetric problems, these results
are in often surprising agreement with computation$^{\cite{AC94,
AP95B, MHBCKLLNPS97, APPSS95, AP95}}$.  See also work on
perturbative fields on fixed backgrounds$^{\cite{KLP97}}$.  Three-dimensional
black hole work has progressed with maximal slicing in
3-d$^{\cite{ACMSSTT95, ABsub1996}}$.  However, it became clear that an
Eddington-Finkelstein adapted coordinate system is preferable because the
representation of a single black hole is static in that case.

At present we have succeeded in long-term single black hole evolution
for both nonrotating (Schwarschild) an drotating (Kerr) black holes. By
imposing blending boundary conditions to analytically known solutions,
we can in some situations run the code \textit{forever}.  Norms of
changes of field variables across the grid decrease to, and remain at,
machine round-off levels. \cite{Randy} We are investigating the range
of parameters that allows this result. We have also demonstrated a
black hole moving across the grid for periods up to
$60M$$^{\cite{Alliance97b}}$, and shorter times for moving spinning
black holes \cite{Randy}.  Further, we have been able to demonstrate
brief evolutiuons of interacting spinning black holes moving through
the computational grid \cite{Randy}.  Apparent horizon ``locking" and
is necessary for better performance. This means that we will
activelytrack the apparent horizon and fix coordinates to it.  We have
developed three separate approaches:  expansion in spherical
harmonics;$^{\cite{ABBLMSW95H, ADMSS95H, ACLMSS96, LMSS95, BCSST96}}$
direct solution of the apparent horizon elliptic equation,$^{\cite{Huq
1996}}$ and curvature flow methods$^{\cite{SHM1997}}$ to the tracking
problem, and we are enabling horizon tracking for new runs.

Essential to simulations of gravitational waveforms is the ability to
extract waves at infinity. We have developed a method based on
matching to a perturbative outer module,$^{\cite{AP95, AAE96,
Alliance97a}}$ and a matching to characteristic
evolution$^{\cite{PL95, HM94, GLPWXX, BGHMPW96,
BGHMPW97, BGLW 97, GLPWYY}}$.  Various technical aspects
of the matching approaches require computation or integration on the
matching sphere, and these have been developed$^{\cite{GPW-Gro 95a,
GPW-Gro 95b, BGLW98}}$.

A substantial effort was made to rewrite the Einstein system in a way that
more closely presented its hyperbolic nature. This work proceeded on two
fronts. A scheme which is based on the Arnowitt-Deser-Misner formalism, but
which introduces additional spatial variables, was introduced by Bona and
Masso,$^{\cite{BM92}}$ and developed substantially$^{\cite{BSMS95},
\cite{BMSS95, BSMS95b}}$.  A new formulation, which deals with
one higher time derivative of the metric, and demonstrated explicit
hyperbolicity, was also developed$^{\cite{9, AAC-BY95, AAE96,
AAC-BY95b, AAC-BY96, AY95}}$.  This latter
formalism has had significant impact outside relativity, and has been
studied in the context of more general field theory.

A number of more or less general reports have been given describing
the Binary Black Hole Grand Challenge Alliance$^{\cite{RM94,
ES94, ES95, ES96, LSF96, SLS96, PL97, ACMSSTT95}}$
and other work supported/stimulated by the
Alliance$^{\cite{L96}}$.

Cray Research used figures from Grand Challenge Alliance research on
the cover of its 1997 University Research and Development Grant
Report.

\begin{center}
\textbf{Human Resource Development}
\end{center}


Our group has a large and growing number
of graduate students who share our enthusiasm regarding the future of
computational science. These students receive an outstanding
grounding in computational physics, numerical analysis and systems-development
skills, and a continued high level of graduate training is a primary goal
of our overall research program.   They contribute substantially
to the research program.

A very large number of students and postdocs were involved in the Binary
Black Hole Grand Challenge. Those from Texas supported by the Alliance or
by other NSF funding to Matzner and Choptuik were:
Matthew Choptuik (postdoc, now Associate professor of Physics),
Robert Marsa  (PhD 1995, now a post-doctoral fellow in the Center),
Dale Choi (PhD expected 1997), Steven Liebling (PhD
expected 1997), Randall Correll (PhD expected in 1999),
Mark Dubal (postdoc), Tim Galavan (graduate student),
Reid Guenther (PhD 1995),  Eric Hirschman (postdoc),
Paulo Holvorcem (postdoc),  Mijan Huq (PhD 1996, now a postdoc in the Center),
Scott Klasky (PhD 1994, now a postdoc at Syracuse),
John LaChapelle (postdoc),
Juan Lara (graduate student),
Manish Parashar (postdoc),
Premana Premadi (PhD 1996),
Jonathan Thornburg (postdoc),  Robin Tuluie (PhD 1993)
David Neilsen (graduate student), Scott Hawley (graduate student) Chun
Yen Wang (graduate student), Carmen Molina-Paris(graduate student),
Jodie Martin (undergraduate student), Mike Hicks (undergraduate
student),
Johnathan Thornburg (postdoc).



Results of Prior Research Sponsored by NSF

Robert van de Geijn

Grant:  CMS-9522964
Principle Investigators: Greg Rodin (PI), J.C. Browne (Co-PI), 
          J. Tinsley Oden (Co-PI), Robert van de Geijn (Co-PI)
Period: April 1, 1995 - March 31, 1998
Amount: \$1,200,000
Title:  Application of High Performance Computing to Micromechanical
        Characteristics of Composite Materials

This grant has supported 4 Ph.D. students (1 has completed his Ph.D.,
1 will defend in Aug. 1998, and 2 will finish within the next two
years), 1 MS student, and 3 post-doctoral students.  In addition,
the project supported a visiting scientist from the People's Republic
of China.

The principal results obtained during the project are as follows:

Fast Boundary Element Methods:  We have developed FLEMS, a 3-D 
boundary element code for rapid solution of many-particle problems
in linear elasticity \cite{Fu_et_al_98,Fu_98}  This code allows
the integration of any boundary element method with any iterative
strategy that includes a fast summation method.  The present version
includes the standard collocation boundary elemetn mehthod and a
GMRES solver in which matrix-vector multiplication is performed
with the fast multipole method.  FLEMS is the first code of its
kind for solving multiscale materials problems.

Fast Dislocation Dynamics Methods:  Following theoretical ideas
developed in connection with FLEMS, we have developed a similar 
approach and code for dislocation dynamics simulations \cite{Rodin_98, Fu_98}.
This code is being adapted by researchers at LLNL, ALCOA and
in France.

Modeling Error Estimates: We have introduced and analyzed the concept
of modeling errors.  These errors are associated with replacing a 
fine-scale model with a coarse-scale model.  Clearly this displacement
procedure is central to multiscale problems.  Main results have been 
published by Zohdi et al. \cite{Zohdi} and Oden and Zohdi \cite {Oden+Zohdi_97}.

Domain Decomposition Methods: We have developed two complementatry
domain decomposition methods for multiscale problems.  The first
method \cite{Zohdi, Oden+Zohdi_97}.
is based on
hierarchical partitioning of the domain.  The second \cite{Podnos_98}
allows one to represent complex microstructures as a combination
of problems that involve individual miscrostructural features whose
interactions are computed using a structured mesh.



\begin{thebibliography}{99}
        \bibitem{1}  http://www.npac.syr.edu/projects/bh/
        Click on ``New Developments".
        Multiple interacting black holes
        are an active research project, and funding for
        the continuation of this work is requested
        under this proposal. Results will be posted
        to this website as they are developed. \\

 \bibitem{SCIVIS} http://kopernik.npac.syr.edu:8888/scivis/
        \bibitem{2} http://godel.ph.utexas.edu/)(link GC Toolkit).

\bibitem{SCIVIS2} B. Ki, S. Klasky,
        ``Collaborative Scientific Visualization''
        \textit{Concurrency: Practice and Experience}, (Nov, 1997).

        \bibitem{3} R. Matzner, H.E. Seidel, S. Shapiro, L. Smarr, W-M. Suen,
        S. Teukolsky, J. Winicour, ``Geometry of a Black Hole Collision,''
        \textit{Science}, (Oct, 1995).

        \bibitem{4}   S.L. Shapiro, S.A. Teukolsky and J. Winicour,
        ``Toroidal Black Holes and Topological Censorship,'' \textit{Phys.
         Rev. D} {\bf 52}, 6982 (1995).

        \bibitem{5}
         http://kopernik.npac.syr.edu:8888/scivis/index.html

        \bibitem{6} http://www.ticam.utexas.edu/parashar/public\_html
        /DAGH/ \\ http://www.npac.syr.edu/projects/bbh/DAGH/ \\
        M. Parashar and J.C. Browne, ``System Engineering for High
        Performance Computing Software: The HDDA/DAGH Infrastructure
        for Implementation of Parallel Structured Adaptive Mesh
        Refinement,'' to be published in the Structured Adaptive Mesh
        Refinement Grid Methods, IMA Volumes in Mathematics and its
        Applications, Springer-Verlag.  \\
        M.~Parashar and J.C.~Browne,``Systems Engineering Issues in
        the Implementation of an Infrastructure for Parallel Structure
        Adaptive Meshes,'' Invited paper, Workshop of Structured
        Adaptive Mesh Refinement Grid Methods, SIAM Conference on Parallel
        Processing for Scientific Computing, March, 1997. \\
        M.~Parashar and J.C.~Browne,``Object-Oriented Programming
        Abstractions for Parallel Adaptive Mesh-Refinement,'' Parallel
        Object-Oriented Methods and Applications (POOMA), February, 1996. \\
        M.~Parashar and J.C.~Browne, ``On Partitioning Dynamic Adaptive
        Grid Hierarchies,'' Proceedings of the $\rm 29^{th}$ Annual
        Hawaii International Conference on System Sciences, January,1996. \\
        M.~Parashar and J.C.~Browne,``Distributed Dynamic Data-Structures
        for Parallel Adaptive Mesh-Refinement,'' Proceedings of the
        International Conference for High Performance Computing, December,1995.

 \bibitem{SBCST97} ``Numerical Evolution of Black Holes with a
Hyperbolic Formulation of General Relativity", M. A. Scheel,
T. W. Baumgarte, G. B. Cook, S. L. Shapiro,
and S. A. Teukolsky, submitted to Phys. Rev D

\bibitem{CS97} ``Well-behaved harmonic tome slices of charged, rotating,
boosted black hole",
   G. Cook, M. Scheel, submitted to Phys. Rev D (1997)

        \bibitem{8}  http://www.npac.syr.edu/ \\
        T. Haupt, ''Fortran 90 Implementation of the Berger-Oliger
        Algorithm for Adaptive Mesh Refinement,'' Status Report-02.\\
        N. Chrisochoides, G. Fox, T. Haupt, ''A Computational Tookit
        for Colliding Black Holes and CFD,'' SCCS-626, in proceedings
        of AAIA Conference, (June, 1994). \\
        T. Haupt, M. Choptuik, ``Definition of Fortran90 data structures and
        an example program that uses them.''\\
        T. Haupt, ``Fortran 90 Implementation of the Adaptive Mesh
        Refinement Algorithm,''
        http://www.npac.syr.edu/project/bbh/AMR/AMR-02.ps \\
        M. Miller, T. Haupt,``Gravitational Wave Extraction -
        A Benchmark?'' SCCS-681.

        \bibitem{9}  Y.Choquet-Bruhat and J.W. York, ``Geometrical Well
        Posed Systems for the Einstein Equations,'' C.R. Acad. Sci. 
        Paris, \textbf{321} 1089 (1995).

         \bibitem{10} A. Abrahams, A. Anderson, Y. Choquet-Bruhat, J. W. York
        Jr., ``Geometrical Hyperbolic Systems for General Relativity and Gauge
        Theories,''Phys Rev Lett. \textbf{75} 3377 (1995).\\
          ``A non-strictly hyperbolic system for the
        Einstein equations with arbitrary lapse and shift,'' gr-qc/9607006.

        \bibitem{11}  Nigel T. Bishop, Roberto Gomez, Paulo R. Holvorcem,
         Richard A. Matzner, Philippos Papadopoulos, Jeffrey Winicour.
            ``Cauchy-Characteristic Evolution and Waveforms,''
         Journal of Computational Physics, Volume 136, Number 1, pg.140-167,
        (September 1, 1997).

        \bibitem{12}
        http://wwwrel.ph.utexas.edu/Members/mijan/GC/June97/june.html
        %\bibitem{xx }  ``A non-strictly hyperbolic system for the Einstein
        % with arbitrary lapse and shift,'' gr-qc/9607006.


        \bibitem{13}  http://wwwrel.ph.utexas.edu/~mijan/blackhole/ \\


\bibitem{Anninos94e} P. Anninos, G. Daues, J. Mass\'o, E. Seidel, and
   W-M. Suen, ``Horizon boundary conditions for black hole spacetimes,"
   \textit{Phys. Rev. D} {\bf 51}, 5562 (1995).

\bibitem{cook97} M.A. Scheel, T.W. Baumgarte, G.B. Cook, S.L. Shapiro
    S.A. Teukolsky, ``Numerical Evolution of Black Holes with a Hyperbolic
    Formulation of General Relativity," gr-qc/9708067   %16

        \bibitem{14} % Alliance paper (Texas/IGPP Workshop Presentation, Feb.
       % 3, 4, 1997 \\ (http://www.npac.syr.edu/projects/bh/) \\
        %(http://northstar.ncsa.uiuc.edu:10000/homepages/lr/adm.html).
        The Binary Black Hole Grand Challenge Alliance, ``Gravitational wave
        extraction and outer boundary conditions by perturbative matching,''
        submitted to Phys. Rev. Let. (1997).

        \bibitem{15}   R.A. Matzner, ``Null Adapted Data and Coordinate
        Conditions for Multiple Black Holes,'' Texas/IGPP Workshop
        Presentation, Feb. 3,4, 1997 \\ (http://www.npac.syr.edu/projects/bh/).

     % \bibitem{}  http://www.npac.syr.edu/projects/bh/).

         \bibitem{16} R.P. Kerr and A. Schild, ``Some Algebraically
          Degenerate Solutions of Einstein's Gravitational Field
          Equations,'' \textit{Applications of Nonlinear Partial
          Differential Equations in Mathematical Physics}, Proc. of
          Symposia b Applied Math., Vol. XV11, (1965).\\
         ``A New Class of Vacuum Solutions of the Einstein Field
          Equations,'' \textit{Atti del Convegno Sulla Relativita
          Generale: Problemi Dell'Energia E Onde Gravitazionale},
          G. Barbera,ed.,(1965).  %19
\bibitem{Ch94}  ``An Implementation of the Berger-Oliger Mesh
Refinement Algorithm for the Wave Equation in Spherical Symmetry,''
Matthew W. Choptuik, technical report (March 1994).

\bibitem{MSW95}  ``Adaptive Mesh Refinement in Numerical
Relativity,'' J. Masso, E. Seidel, and P. Walker, in \textit{General
Relativity} (MG7 proceedings), R. Ruffini and M. Keiser (eds.), World
Scientific (1995).

\bibitem{H95}  ``Fortran 90 Implementation of the Berger-Oliger
Algorithm for Adaptive Mesh Refinement,'' Tomasz Haupt (1995).%22

%\bibitem{refs??}  refs?? old [23]  Should be [9]

\bibitem{CFH94}  ``A Computational Toolkit for Colliding Black Holes
and CFD,''  N.P. Chrisochoides, G.C. Fox, and T. Haupt (1994). %23

\bibitem{NC95}  ``Multithread model for dynamic load balancing
parallel adaptive PDE computations,'' Nikos Chrisochoides, to appear
in Applied Numerical Mathematics Journal (1995).

\bibitem{PB95}  ``Distributed Dynamic Data-Structures for Parallel
Adaptive Mesh-Refinement,'' M. Parashar and J.C. Browne, in
\textit{Proceedings of the International Conference for High Performance
Computing} (December 1995).

\bibitem{PB96}  ``Object-Oriented Programming Abstractions for
Parallel Adaptive Mesh-Refinement,'' M. Parashar and J.C. Browne, in
\textit{Parallel Object-Oriented Methods and Applications (POOMA)}
(February 1996).

\bibitem{PB96b} ``On Partitioning Dynamic Adaptive Grid Hierarchies,''
M. Parashar and J.C. Browne, in \textit{Proceedings of the 29th Annual
Hawaii International Conference on System Sciences} (January 1996)
(IEEE).

\bibitem{PB97}  ``System Engineering for High Performance Computing
Software:  The HDDA/DAGH Infrastructure for Implementation of Parallel
Structured Adaptive Mesh Refinement,'' M. Parashar and J.C. Browne, to
be published in the \textit{Structured Adaptive Mesh Refinement Grid
Methods, IMA Volumes in Mathematics and its Applications},
Springer-Verlag.

\bibitem{PB97b}  ``Systems Engineering Issues in the Implementation of
an Infrastructure for Parallel Structured Adaptive Meshes,'' M.
Parashar and J.C. Browne, invited paper, Workshop in Structured
Adaptive Mesh Refinement Grid Methods, SIAM Conference on Parallel
Processing for Scientific Computing (March 1997).

\bibitem{PBEK97}  ``A Common Data Management Infrastructure for
Parallel Adaptive Algorithms for PDE Solutions,'' M. Parashar, J.C.
Browne, C. Edwards, and K. Klimkowski, accepted for presentation at
Supercomputing 97 (November 1997). %0ld 31. New 30

%\bibitem{references?}  should be 6 old 32

\bibitem{WYWADPS97}  ``A New Generation EOS Compositional Reservoir
Simulator:  Formulation and Discretization,'' P. Wang, I. Yotov, M.
Wheeler, T. Arbogast, C. Dawson, M. Parashar, and K. Sepehrnoori, in
\textit{Proceedings of the Society of Petroleum Engineers, Reservoir
Simulation Symposium} (June 1997). %new 32 old34

\bibitem{PWPWW97}  ``A New Generation EOS Compositional Reservoir
Simulator:  Framework and Multiprocessing,'' M. Parashar, J.A.
Wheeler, G. Pope, K. Wang, and P. Wang, in \textit{Proceedings of the
Society of Petroleum Engineers, Reservoir Simulation Symposium} (June 1997).

\bibitem{34}  ``Integrated Data-Management for Computational
Steering,'' M. Parashar and J.C. Browne, submitted for publication to
the 31st Annual Hawaii International Conference on System Sciences
(January 1998).

\bibitem{Randy}  ``Computational Evolution of Ninary Black Hole Spacetimes'' 
R. Correll, PhD dissertation, The university of Texas at Austin (1998)

\bibitem{35} R.L. Marsa and M.W. Choptuik, ``Black Hole-Scalar Field
Interactions in Spherical Symmetry,'' Physical Review D, \textbf{54}
4929-4943 (1996)  %Seidel,SuenHorizonTracking old 36  new 34
%In this Chooptuik paper find ref to seidel,suen

\bibitem{SST95}  ``Collapse to Black Holes in Brans-Dicke Theory:
I.  Horizon Boundary Conditions for Dynamical Spacetimes,'' Mark A.
Scheel, Stuart L. Shapiro, and Saul A. Teukolsky, \textit{Physical
Review} \textbf{D51} 4208 (April 1995).

\bibitem{SST95B}  ``Collapse to Black Holes in Brans-Dicke Theory:
II.  Comparison with General Relativity,'' Mark A. Scheel, Stuart L.
Shapiro, and Saul A. Teukolsky,  Physical Review  \textbf{D51}{Physical Review}
 \textbf{D51}
4236 (April 1995).

\bibitem{37} A.M. Abrahams and C.R. Evans,``Universality in Axisymmetric
   Vacuum Collapse'', Physical Review  \textbf{D51} 3998-4003
   (1994) %old 39  n37

\bibitem{S+T92} S.L. Shapiro and S. Teukolsky,
\textit{Physical Review} \textbf{45} 2739 (1992).

\bibitem{39} H. Kurki-Suonio, P. Laguna and Richard A. Matzner,
``Inhomogeneous Inflation: Numerical Evolution'',  Physical Review
\textbf{D47} 3611 (1993). %Pull out NCA - ADM  pull out internally

\bibitem{GPW94}  ``Null Cone Evolution of Axisymmetric Vacuum
Spacetimes,'' R. Gomez, P. Papadopoulos, and J. Winicour,
\textit{J. Math. Phys.} \textbf{35} 4184 (1994).

\bibitem{L96}   P. Laguna,``Alternatives to Finite Difference Methods in
Numerical Relativity,'' in \textit{Relativity and
Scientific Computing}, eds. F.W. Hehl, R.A. Puntigam, and H. Ruder
(Springer/Verlag Berlin, 1996). 

\bibitem{ACMSSTT95} ``3-D Numerical Relativity at NCSA,'' P. Anninos,
K. Camarda, J. Masso, E. Seidel, W-M Suen, M. Tobias, and J. Towns, to
appear in \textit{General Relativity} (MG7 proceedings), eds. R.
Ruffini and M. Keiser, World Scientific (1995). %n41

\bibitem{BGPWX96}  ``Steps in the construction of an extraction
module,'' Nigel T. Bishop, Roberto Gomez, Philippos Papadopoulos, and
Jeffrey Winicour, \textit{Pitt-Penn State-UniSA BBH Report} (1996).

\bibitem{GPWS97}  ``Stereographic Coordinate Patches,'' R. Gomez, P.
Papadopoulos, and J. Winicour, BBH working paper -- program listing
also available (1997).

\bibitem{GPSWE97}  ``Computational application of the eth formalism,''
R. Gomez, P. Papadopoulos, and J. Winicour, in preparation (1997).

\bibitem{E+C1994} ``Observation of Critical Phenomena and
Self-Similarity in the Gravitational Collapse of Radiation Fluid,''
C.R. Evans and J.S. Coleman, in \textit{Phys. Rev. Lett.} \textbf{72},
1782 (1994).

\bibitem{46}  Matthew W. Choptuik, ``Universality and Scaling in
Gravitational Collapse of a Massless Scalar Field,'' Physical
Review Letters \textbf{70}  9-12 (1993). %old 48


\bibitem{AST95} ``Calculation of Gravitational Wave Forms From Black
Hole Collisions and Disk Collapse:  Applying Perturbation Theory to
Numerical Spacetimes,'' Andrew M. Abrahams, Stuart L. Shapiro, and Saul
Teukolsky, \textit{Physical Review} \textbf{D51} 4295 (April 15, 1995).

\bibitem{Ch94B} ``Critical Behaviour in Scalar Field Collapse,''
Matthew W. Choptuik, in \textit{Deterministic Chaos in General
Relativity}, eds. D. Hobill et al, Plenum Press, New York (January
1994).

\bibitem{Ch95} ``Critical Phenomena in Gravitational Collapse,''
Matthew W. Choptuik, to appear in \textit{Proc. XI Congress
Mathematical Physics}, D. Iagolnitzer (ed.) (1995).

\bibitem{MC96} ``Black Hole-Scalar Field Interactions in Spherical
Symmetry,'' Robert L. Marsa and Matthew W. Choptuik, gr-qc/9607034
(1996).

\bibitem{CCB96}  ``Critical Behaviour in Gravitational Collapse of a
Yang-Mills Field,'' Matthew W. Choptuik, Tadeusz Chmaj, and Piotr
Bizon, gr-qc/9603051 (1996).

\bibitem{52} G.B. Cook, M. W. Choptuik, M. R. Dubal, S. Klasky, Richard
A. Matzner and S.R. Oiveira, ``Three-Dimensional Initial Data for the
Collision of Two Black Holes,'' Phys.Rev. \textbf{D47 } 1471 (Feb.1993).
%old 54

\bibitem{C94II} ``Three-dimensional initial data for the collision of
two black holes II:  Quasi-circular orbits for equal-mass black
holes,'' Gregory B. Cook (submitted for publication).

\bibitem{ADMSS95} ``Singularity Avoidance in Numerical Black Hole
Spacetimes,'' Peter Anninos, Greg Daues, Joan Masso, Edward Seidel,
Wai-Mo Suen, \textit{General Relativity} (MG7 Proceedings) Ruffini
and Keiser (eds.), World Scientific (1995).

\bibitem{BS94a} ``The Evolution of Distorted Rotating Black Holes I:
Methods and Tests,'' Steven R. Brandt and Edward Seidel,
\textit{Physical Review} \textbf{D52} 856 (July 15, 1995).

\bibitem{BS94b} ``The Evolution of Distorted Rotating Black Holes
II:  Dynamics and Analysis,'' Steven R. Brandt and Edward Seidel,
\textit{Physical Review} \textbf{D52} 856 (July 15, 1995).

\bibitem{BS95} ``Rotating Black Hole Spacetimes,'' S. Brandt and E.
Seidel, to appear in \textit{General Relativity} (MG7 Proceedings),
R. Ruffini and M. Keiser (eds.), World Scientific (1995).

\bibitem{AHSSS93} ``The collision of Two Black Holes,'' Peter Anninos,
David Hobill, Edward Seidel, Larry Smarr, and Wai-Mo Suen,
\textit{Physical Review Letters} \textbf{71} 2851 (1993).

\bibitem{AHSSS94} ``The Head-On Collision of Two Equal Mass Black
Holes,'' Peter Anninos, David Hobill, Edward Seidel, Larry Smarr, and
Wai-Mo Suen,  \textbf{D52} 2044  (1995).

\bibitem{AHSSS94T} ``The Head-On Collision of Two Equal Mass Black
Holes:  Numerical Methods,'' Peter Anninos, David Hobill, Edward
Seidel, Larry Smarr, and Wai-Mo Suen, \textit{NCSA Technical Report
24} (1994).

\bibitem{MSSSSTW95} ``Geometry of a Black Hole Collision,'' Richard
A. Matzner, H.E. Seidel, S.L. Shapiro, L. Smarr, W-M Suen, Saul A.
Teukolsky, and J. Winicour, \textit{Science} \textbf{270} 941 (1995)
(cover article).

\bibitem{M95} ``Numerical Relativity and Black Hole Horizons,'' Joan
Masso, in \textit{Inhomogenous Models and other topics} (ERE94), A.
Molina and J.M. Senovilla (eds.), World Scientific (1995).

\bibitem{ADBLMSSSW95} ``Event Horizons of Numerical Black Holes,'' P.
Anninos, D. Bernstein, S. Brandt, J. Libson, J. Masso, E. Seidel, L.
Smarr, W-M Suen, and P. Walker, to appear in \textit{General
Relativity} (MG7 proceedings), R. Ruffini and M. Keiser (eds.), World
Scientific (1995).

\bibitem{LMSSW96} ``Event Horizons in Numerical Relativity:  Methods
and Tests,'' J. Libson, J. Masso, E. Seidel, W-M Suen, and P. Walker,
\textit{Physical Review} \textbf{D53} 4335-4350 (1996).

\bibitem{AC94} ``Collisions of Boosted Black Holes:  Perturbation
Theory Prediction of Gravitational Radiation,'' Andrew M. Abrahams
and Gregory B. Cook, \textit{Physical Review} \textbf{D50} R2364
(August 15, 1994).

\bibitem{AP95B}``Black-hole collisions from Brill-Lindquist initial
data:  Predictions of perturbation theory,'', A. Abrahams, R.H. Price,
\textit{Physical Review} \textbf{D53-4} 1972 (1996).

\bibitem{MHBCKLLNPS97} ``Analysis of `Gauge Modes' in Linearized
Relativity,'' Richard A. Matzner, Mijan Huq, Alonso Botero, Dae Il
Choi, Ullar Kask, Juan Lara, Steven Liebling, David Neilsen, Premana
Premadi, and Deirdre Shoemaker, \textit{Classical and Quantum Gravity}
\textbf{14} L21 (1997).

\bibitem{APPSS95} ``Head-On Collision of Two Black Holes:  Comparison
of Different Approaches,'' P. Anninos, R. Price, J. Pullin, E. Seidel,
and W-M Suen (June 1995).

\bibitem{AP95} ``Applying black hole perturbation theory to
numerically generated space-times,'' A. Abrahams, \textit{Physical Review}
\textbf{D53-4} 1963 (1996). %69  old 71

\bibitem{Alliance97b} ``Boosted Black Hole Evolutions with 
Singularity Excision,'' 
The Binary Black Hole Grand Challenge Alliance, \textit{Phys.
Rev. Lett.} (submitted 1997). % [71] old 74

\bibitem{ABBLMSW95H} ``Dynamics of Apparent and Event Horizons,'' Peter
Anninos, David Bernstein, Steven Brandt, Joseph Libson, Joan Masso,
Edward Seidel, Larry Smarr, Wai-Mo Suen, and Paul Walker,
\textit{Physical Review Letters} \textbf{74} 630 (January 1995).

\bibitem{ADMSS95H} ``Horizon Boundary Condition for Black Hole
Spacetimes,'' Peter Anninos, Greg Daues, Joan Masso, Edward Seidel,
and Wai-Mo Suen, \textit{Physical Review} \textbf{D51} 5562 (May 15,
1995).

\bibitem{ACLMSS96} ``Finding Apparent Horizons in Dynamic 3D Numerical
Spacetimes,'' submitted to \textit{Physical Review D} (1996).

\bibitem{LMSS95} ``A 3-D Apparent Horizon Finder,'' J. Libson, J.
Masso, E. Seidel, and W-M Suen, in \textit{General Relativity} (MG7
proceedings), R. Ruffini and M. Keiser (eds.), World Scientific (1995).

\bibitem{BCSST96} ``Implementing an apparent-horizon finder in three
dimensions,'' T.W. Baumgarte, G.B. Cook, M.A. Scheel, S.L. Shapiro, S.A.
Teukolsky, gr-qc/9606010 (June 6, 1996).

\bibitem{Huq 1996} ``Apparent Horizon Location in Numerical
Spacetimes,'' Mijan Huq, PhD dissertation, University of Texas at
Austin (1996). %old 80

\bibitem{SHM1997} ``Curvature Flow Methods and Apparent Horizons",
Deirdre Shoemaker, Mijan F. Huq, and Richard A. Matzner, in preparation
(1997).

\bibitem{AAE96} ``Gravitational waveform extraction and outer boundary
conditions for hyperbolic numerical relativity simulations,'' A.
Abrahams, A. Anderson, and C.R. Evans, Preprint (June 28, 1996).

\bibitem{Alliance97a} ``Schwarzschild-perturbative gravitational
wave extraction and outer boundary conditions,'' The Binary Black Hole
Grand Challenge Alliance, submitted to \textit{Physical Review
Letters} (1997).

\bibitem{PL95} ``Cauchy-Characteristic evolution of scalar waves in
curved spacetime.  I.  Static, spherically symmetric spacetimes,'' P.
Papadopoulos, P. Laguna (1995).

\bibitem{HM94} ``Pittsburgh Group Code:  Gravitational Wave
Extraction -- A Benchmark'' (evolution of scalar fields from
characteristic data,'' Tomasz Haupt and Mark Miller (October 1994).l

\bibitem{GLPWXX} ``Cauchy-characteristic Evolution of
Einstein-Klein-Gordon Systems,'' R. Gomez, P. Laguna, P. Papadopoulos,
J. Winicour.

\bibitem{BGHMPW96} ``Cauchy-characteristic matching:  A new approach
to radiation boundary conditions,'' Nigel T. Bishop, Roberto Gomez,
Paulo R. Holvorcem, Richard A. Matzner, Philippos Papadopoulos, and
Jeffrey Winicour, \textit{Physical Review Letters} \textbf{76} 4303
(1996).

 \bibitem{gom97}
 R. G\'{o}mez, R.  Marsa and J. Winicour, ``Black hole excision with
  matching'', \textit{Phys. Rev. D} \textbf{56}, (Nov.
 1997), gr-qc/9708002. \\


\bibitem{BGHMPW97} ``Cauchy-Characteristic Evolution and Waveforms,''
N.T. Bishop, R. Gomez, P.R. Holvorcem, R.A. Matzner, P. Papadopoulos, J.
Winicour,  \textit{J. Comp. Phys.} \textbf{136} 140 (1997).

\bibitem{BGLW 97} Nigel T. Bishop,Roberto Gomez,Luis Lehner,
Jeffrey Winicour, `` Cauchy-characteristic extraction in numerical
relativity,'' to appear in Phys. Rev. D (1997). %BGLW 97 old[90]

\bibitem{GLPWYY} ``Cauchy-characteristic evolution of
Einstein-Klein-Gordon Systems,'' Roberto Gomez, Pablo Laguna,
Philippos Papadopoulos, and Jeffrey Winicour, to appear in
\textit{Physical Review D}.

\bibitem{GPW-Gro 95a} ``Finite Differencing on the Sphere,'' R.
Gomez, P. Papadopoulos, and J. Winicour, \textit{Proceedings of the
Seventh Marcel Grossman meeting} (1995).

\bibitem{GPW-Gro 95b} ``Computational Application of the eth
formalism,'' Roberto Gomez, Philip Papadoupolos, J. Winicour, to
appear in \textit{Proceedings of the VII Marcel Grossman meeting}, R.
Ruffini and M. Reiser (eds.), World Scientific (1995).

\bibitem{BGLW98} ``Gravitational Instability of a Kink,'' Williams
Barreto, Roberto Gomez, Luis Lehner, and Jeffrey Winicour, to appear
in \textit{Physical Review D} (1998). % old 94

\bibitem{BM92} C.Bona and J. Masso, Phys. Rev. Lett. \textbf{68}
 1097 (1992).

\bibitem{BSMS95} ``A Class of Hyperbolic Gauge Conditions,'' C. Bona,
J. Stela, J. Masso, and E. Seidel, in \textit{General Relativity} (MG7
proceedings), R. Ruffini and M. Keiser (eds.), World Scientific (1995).

\bibitem{BMSS95} ``New Formalism for Numerical Relativity,'' Carles
Bona, Joan Masso, Edward Seidel, and Joan Stela, \textit{Physical
Review Letters} \textbf{75} 600 (July 1995).

\bibitem{BSMS95b} ``Invariant Algebraic Slicing of the Spacetime,''
C. Bona, J. Stela, J. Masso, and E. Seidel, in \textit{Inhomogenous
Models and other topics} (ERE94), A. Molina and J.M. Senovilla
(eds.), World Scientific (1995).

\bibitem{AAC-BY95} ``Einstein and Yang-Mills theories in hyperbolic
form without gauge-fixing,'' A. Abrahams, A. Anderson, Y.
Choquet-Bruhat, and J.W. York Phys. Rev. Lett.
\textbf{75} 3377 (1995).

\bibitem{KLPA97}``Dynamics of perturbations of rotating black holes", 
w. Krivan, P. Laguna, P. Papadopoulos, and Nils Andersson, Phys. 
Rev. D\textbf{56}, 3395 (1997)

\bibitem{BCKP1997}P.Brady, C. Chambers, W. Krivan and P. Laguna, ''Telling Tails in the
Presence of a Cosmological Constant,'' submitted to Phys. Rev. D (1996).

\bibitem{KLP97}W. Krivan, P. Laguna and P. Papadopoulos, ''Dynamics of Scalar Fields in
the Background of Rotating Black Holes,''  Phys. Rev.\textbf{D56}
(Sept.1997).

\bibitem{KLP97B}W. Krivan, P. Laguna and P. Papadopoulos, ''Dynamics of Spin-2 Fields in
Kerr Background-geometries,'' to appear Proceedings Texas Symposium for
Relativistic Astrophysics  (1997).

\bibitem{AAC-BY95b} ``Geometrical Hyperbolic Systems for General
Relativity and Gauge Theories,'' A. Abrahams, A. Anderson, Y.
Choquet-Bruhat, and J. York, Jr. submitted to \textit{Classical and
Quantum Gravity} (1995). %old 100

\bibitem{AAC-BY96} ``A non-strictly hyperbolic system for the
Einstein equations with arbitrary lapse and shift,'' A. Abrahams, A.
Anderson, Y. Choquet-Bruhat, and J. York, Jr., gr-qc/9607006 (1996).

\bibitem{AY95} ``3+1 general relativity in hyperbolic form,'' A.
Abrahams, J.W. York, Jr., paper to appear in Proceedings of Les
Houches conference (1995).

\bibitem{RM94}  ``Computing Gravitational Radiation from the
Collision of Black Holes:  Status Report,'' Richard Matzner in
\textit{Phenomenology of Unification from Present to Future}, G.
Diambrini-Palazzi et al (eds.), World Scientific (1994).

\bibitem{ES94}  ``Solving Einstein's Equations on Supercomputers,''
Edward Seidel (1994).

\bibitem{ES95}  ``Numerical Relativity,'' E. Seidel, in proceedings
of ICGC 95, Kluwer Press, to appear (1995).

\bibitem{ES96}  ''New Developments in Numerical Relativity,'' E.
Seidel, in \textit{Helvetica Physica Acta}, to appear (1996).

\bibitem{LSF96}  ``A Numerical Approach to Binary Black Hole
Coalescence,'' L.S. Finn, to appear in proceedings of the GR14
conference, gr-qc/9603004 (1996).

\bibitem{SLS96}  ``Calculating Gravitational Waveforms -- By Any
Means Necessary!,'' S.L. Shapiro, invited talk to appear in
\textit{Proceedings of International Conference on Gravitational
Waves:  Sources and Detectors}, Cascina (Pisa), Italy (March 19-23,
1996).

\bibitem{PL97}  ``Black Hole Collisions:  A Computational Grand
Challenge,'' P. Laguna, in \textit{Gravitational Waves:  Sources and
Detectors}, eds. I. Ciufolini and Francesco Fidecaro, World
Scientific-Singapore (1997).

\bibitem{ABsub1996}P. Anninos, J. Baker, S. Brandt, Hans-P. Nollert,R. Price,J. Pullin,
  and E. Seidel, ''The head-on collsion of close counter-rotating black
  holes,''  submitted to Phys. Rev. D (1996).


\bibitem{A95}  ``Unitarity Restoration in the Presence of Closed
Timelike Curves,'' A. Anderson, \textit{Physical Review} \textbf{D51}
5705 (May 15, 1995).  %old 110

\bibitem{BGLW97}  ``Cauchy-characteristic extraction in numerical
relativity,'' to appear in \textit{Phys. Rev. D} (1997).

\bibitem{C94} Matthew W. Choptuik, ``ad, An Implementation of the
Berger-Oliger Mesh Refinement Algorithm for the Wave Equation in
Spherical Symmetry,'' Technical Report, (March 1994)

\bibitem{LC96} ``Black Hole Criticality in the Brans-Dicke Model,''
Steven L. Liebling and Matthew W. Choptuik, gr-qc/9606057 (1996).

\bibitem{95M} Joan Masso, ``Numerical Relativity and Black Hole
Horizons,'' in \textit{Inhomogeneous Models and Other Topics}
 (ERE94), A. molina and J.M. Senovilla (eds), World Scientific
 (1995).

%\bibitem{AAL 96}  %old 115

\bibitem{C-BY95} ``Geometrical Well Posed Systems for the Einstein
Equations,'' Y. Choquet-Bruhat and J.W. York (August 1995).

\bibitem{AST94}  A.M. Abrahams, S.L. Shapiro and S. A. Teukolsky,
``Disk Collapse in General Relativity,'' Physical Review \textbf{D50}
7282  (Dec.1994).

 \bibitem{17} J. Bowen and J.W. York Jr., ``Two Black Holes
        with Axisymmetric parallel spins: Initial data,'' \textit{
        Class. Quantum Grav.}, \textbf{1},591-610, (1984).

        \bibitem{18}   G.B. Cook, M.W. Choptuik, M. Dubal, S. Klasky,
        R. Matzner and S.R. Oliveira, \textit{Phys. Rev. D},
        1471-1490 (1993).

       \bibitem{19}  Steven Brandt and Bernd Br\"ugmann, '' A simple
        construction of initial data for multiple black holes,''
        \textit{Phys.Rev.Lett.}, \textbf{78} 3606-3609 (1997).
        (http://xxx.lanl.gov/find/gr-qc/9703066).

          \bibitem{20}  Ann S. Almgren, John B. Bell, Phillip Colella, Louis
         H. Howell, Michael L. Welcome, ``A Conservative Adaptive Projection
          Method for the Variable Density Incompressible Navier-Stokes
           Equations,'' submitted to \textit{J. Comp. Phys.} (1997).

        \bibitem{21} P. Anninos, J. Masso, H. E. Seidel, W-M Suen and
        M. Tobias, ``Dynamics of Gravitational Waves in 3D:
        Formulations, Methods, and Tests,''\textit{Phys. Rev. D},
        \textbf{56}, 842-858, (1997).  gr-gc/9610057. \\

 \bibitem{BAL96}
        J.~Balakrishna, G.~Daues, E.~Seidel, W-M.~Suen, M.~Tobias and
        E.~Wang,
        ``Coordinate conditions and their implementation in 3-D numerical
        relativity'',
        LANL preprint gr-qc/9601027 (1996)
\bibitem{ANN95c}
        P.~Anninos, K.~Camarda, J.~Masso, E.~Seidel, W-M.~Suen and J.~Towns,
        ``Three-Dimensional Numerical Relativity:
        The Evolution of Black Holes'',
        {\sl Physical Review D}, {\bf 52}, 2059-2082 (1995)

        \bibitem{22} G. Daues, Ph.D. Dissertation, Washington University,
        (1996).
 \bibitem{CSS1995}``Testing a Simplified Version
of Einstein's Equations for Numerical Relativity,''
G.B. Cook, S.L. Shapiro and S.A. Teukolsky, Preprint. (1995)


        \bibitem{23}  N. T.  Bishop, R. G\'{o}mez, L. Lehner, M. Maharaj
         and J. Winicour, ``High-powered Gravitational News'',
         \textit{Phys. Rev. D}, \textbf{56}, (Nov. 1997), gr-qc/9708065.\\

%        R. Gomez, J. Winicour and R. A. Isaacson, ``Evolution of
%       Scalar Fields from Characteristic Data'',
%      \textit{J. Comp. Phys.} \textbf{98}, 11 (1992).
     \bibitem{222}  Nigel T. Bishop, Roberto Gomez, Paulo R. Holvorcem,
      Richard A. Matzner, Philippos  Papadopoulos and Jeffrey Winicour,
      ``Cauchy-characteristic matching: A new approach to radiation boundary
       condition'', \textit{Phys. Rev. Letters} \textbf{76}, 4303 (1996).

        \bibitem{24} C. Clarke, R. d'Inverno, and J. Vickers,
        \textit{Phys. Rev. D}, \textbf{52}, 6863 (1995).
        M. Dubal, R. d'Inverno, and  C. Clarke,
        \textit{Phys. Rev. D}, \textbf{52}, 6868 (1995).


        \bibitem{25} G. Cook and Scheel; http://xxx.lanl.gov/abs/gr-qc/9708067.

        \bibitem{26}   Bean, Warming, Yee, \textit{J. Comp. Phys.}
        \textbf{48} 200
        (1982).

        \bibitem{27}  http://artemis.phyast.pitt.edu/animations.


\bibitem{P.Anninosetal}P. Anninos et al, Phys. Rev.  \textbf{D50} 3801 (1994).



        \bibitem{28}   http://www.npac.syr.edu/projects/bbh/dagh.html
        \bibitem{TANGO}   http://www.npac.syr.edu/tango
        \bibitem{WebWisdom}   http://www.webwisdom.org/papers/sc97/webwisdom3.html 

        \bibitem{29}   P. D. D'Eath, \textit{Black Holes, Gravitational
        Interactions}, Clarendon, OX. (1996).

        \bibitem{30}   Peter Anninos, Joan Masso, Edward Seidel and Wai-Mo Suen
        ``Numerical Relativity and Black Holes'', Physics World, v. 9, p.
43 (1996)

        \bibitem{31}  Richard H. Price and Jorge Pullin, ''Colliding Black
Holes:
        The Close Limit,'' Phys. Rev. Lett. v. 72, p. 3297 (1994). \\
         Peter Anninos, Richard H. Price, Jorge Pullin, Ed Seidel, Wai-Mo Suen,
         ''Head-on collision of two black holes: comparison of different
         approaches,'' Phys. Rev. D\\ v.52, p. 4462  (1995).
\bibitem{HUQ96}
        M.~Huq,
        ``Apparent Horizon Location in Numerical Spacetimes'',
         The University of Texas at Austin PhD Thesis (1996)
%
\bibitem{BAU96}
        T.~W.~Baumgarte, G.~B.~Cook, M.~A.~Scheel, S.~A.~Teukolsky
        and S.~L.~Shapiro,
        ``Implementing an Apparent Horizon Finder in Three-Dimensions'',
        {\sl Physical Review D}, (in press) (1996)
%
\bibitem{ANN96b}
        P.~Anninos {\sl et al},
        ``Finding Apparent Horizons in Dynamic 3-D Numerical Space
        Times'',
        LANL preprint gr-qc/9609059 (1996)
\bibitem{SEI92a}
        E.~Seidel and W-M.~Suen,
        ``Towards a Singularity Proof Scheme in Numerical Relativity'',
        {\sl Physical Review Letters}, {\bf 69}, 1845-1848 (1992)
%
\bibitem{ALC94}
        M.~Alcubierre and B.~F.~Schutz,
        ``Time-Symmetric ADI and Causal Reconnection: Stable Numerical
        Techniques for Hyperbolic Systems on Moving Grids'',
        {\sl Journal of Computational Physics}, {\bf 112}, 44-77 (1994)
%
\bibitem{ABR94d}
        A.~Abrahams,
        ``The No Boundary Proposal'',
        Talk presented at the NSCA Numerical Relativity Conference,
        (1994)
%
\bibitem{SCH95a}
        M.~A.~Scheel, S.~L.~Shapiro and S.~A.~Teukolsky,
        ``Collapse to Black Holes in Brans-Dicke Theory.
        1. Horizon Boundary Conditions for Dynamical Space-times'',
        {\sl Physical Review D}, {\bf 51}, 4208-4235 (1995)
%
\bibitem{SCH95b}
        M.~A.~Scheel, S.~L.~Shapiro and S.~A.~Teukolsky,
        ``Collapse to Black Holes in Brans-Dicke Theory.
        2. Comparison with General Relativity'',
        {\sl Physical Review D}, {\bf 51}, 4236-4249 (1995)
%
\bibitem{ANN95a}
         P.~Anninos, G.~Daues, J.~Masso, E.~Seidel and W-M.~Suen,
        ``Horizon Boundary Conditions for Black Hole Space-times'',
        {\sl Physical Review D}, {\bf 51}, 5562-5578 (1995)
%

\bibitem{GUE95}
        R.~L.~Guenther,
        ``A Numerical Study of the Time Dependent Shr\"odinger Equation
        Coupled With Newtonian Gravity'',
        The University of Texas at Austin, PhD Thesis (1995)
\bibitem{BRA77}
        A.~Brandt,
        ``Multi-level adaptive solutions to
        boundary-value problems'',
        {\sl Math.~Comp.}, {\bf 31}, 333-390 (1977)
%

\bibitem{ES_H96}E. Seidel, ''New developments in Numerical Relativity,'' in Helvetica
Physica Acta, to apppear, (1996).

\bibitem{RNPL}
http://godel.ph.utexas.edu/Members/marsa/rnpl/refman/refman.html

\bibitem{BRA82}
        A.~Brandt, ``Guide to multigrid development'',
        in {\sl Lecture Notes in Mathematics}, {\bf 960},
        {\sl (Multigrid Methods)},
        W.~Hackbusch and U.~Trottenberg, eds., 220-312 (1982)
%
\bibitem{CHO86b}
        M.~Choptuik and W.~G.~Unruh,
        ``An Introduction to the Multi-Grid Method for Numerical Relativists'',
        {\sl General Relativity and Gravitation}, {\bf 18}, 813--843
        (1986)

\bibitem{CHO95b}
        M.~W.~Choptuik, S.~Klasky and M.~Parashar,
        ``Parallel Multigrid with the DAGH package: Specifications
         and Applications'' (1995)
  http://godel.ph.utexas.edu/Members/matt/Doc/mgspecs.ps

\bibitem{Collins}Tim Collins Ph.D. dissertation, The University of
Texas at Austin, 1995

\bibitem{PLAPACK} 
(van de Geijn, 1997)
Robert A. van de Geijn,
{\em Using PLAPACK: Parallel Linear Algebra Package}, 
The MIT Press, 1997.

\bibitem{MMPI} 
(Edwards, 1996)
H. C. Edwards, {\em MMPI: Asynchronous message management for the 
message-passing interface.}  TICAM Report 96-44, Texas Institute
for Computational and Applied Mathematics, The University of 
Texas at Austin, 1996.

\bibitem{Fu_et_al_98}
Y. Fu, K. Klimkowski, J. Rodin, G. Berger, J.C. Browne, J.C.
Singer, R. van de Geijn, and K.S. Vemaganti, 
"A Fast Solution Method for Three Dimensional
Many-Particle Problems of Linear Elasticity," 
{\em Int. J. Num. Meth. Engrg.},
to appear.

\bibitem{Fu_98}
Y. Fu, "Application of the Fast Multipole Method to Micromechanics,"
Ph.D. Dissertation, The University of Texas at Austin.

\bibitem{Rodin_98}
G.J. Rodin, "Toward Rapid Evaluation of the Elastic Interactions among
Three-Dimensional Dislocations," Phil. Mag. Letters, Vol 77, No. 4, 
pp. 187-190, 1998

\bibitem{Zohdi}
T.I. Zohdi, J.T. Oden, and G.J. Rodin, "Hierarchical Modeling of 
Heterogeneous Bodies," Computer Methods in Applied Mechanics and
Engineering, 138 (1996), pp. 273--298.

\bibitem{Oden+Zohdi_97}
J.T. Oden and T. Zohdi, "Analysis and Adaptive Modeling of
Highly Heterogeneous Elastic Structures," Computer Methods in
Applied Mechanics and Engineering, 148 (1997), pp. 267--381.

\bibitem{Podnos_98}
E.G. Podnos, I. Babuska, and G. J. Rodin, 
"Application of Fictitious Domain Method to Micromechanics of
Non-Linear Composite Materials," submitted to 
{\em Comp. Mech. in Appl. Mech. Engrg.}, 1998.



\end{thebibliography}

\pagebreak
\begin{center}
\textbf{Facilities, Equipment and Other Resources}
\end{center}

This is a proposal for computational research.  In the past few years, the
Center has become quite well-equipped with medium- and high-end
graphics workstations.  This equipment has been, and will continue to
be, absolutely vital to the success of our research.  Our current
facilities include four Silicon Graphics (SGI) R10000 Impact machines
with a total of nearly 1.5 Gbyte main memory and 32 Gbyte of disk
storage, as well as two older SGI Crimson R4000 VGXT machines.
Together, the six R10000/Crimson
machines provide powerful platforms for scientific visualization, code
development, and medium-scale production runs for the PIs, our postdocs,
and (at most) two graduate students.  We also have two entry-level SGI
Indigo and two
SGI Indy machines which are used heavily by graduate students, but which
are rapidly approaching obsolescence and will soon need to be replaced.
We have very good access to campus networks, and adequate access to
national networks and national supercomputer centers.

\pagebreak

\begin{center}
        \textbf{Budget Justification}
\end{center}


This proposal is to support work by Professor M. Choptuik, and 
Professor Richard A. Matzner, 
in computational research on the interaction of black holes. This work will
be done in collaboration with colleagues at The University of Pittsburgh, 
and The Pennsylavania State University, and involves computational science support 
through the University of Syracuse. The budget
principally supports one postdoc, who will be supervised by Matzner and
Choptuik.  It also modestly supports travel, publications, and very necessary 
communication to carry out this collaborative research. 
There is an important piece of
cost sharing: The Center for Relativity commits one postdoc (letter
attached) supported by State of Texas funds as a match to the funded
postdoc position requested here.  This amounts to a \textit{real}
costsharing (salary, fringes, and overhead) of \$354,775 over the five-year
proposed duration of this grant.

\end{document}

