The inspiral of a relativistic binary has played the role of a standard candle
for the first signal to be detected by the gravitational wave observatories
which are now approaching operational readiness. For many years now, this has
spurred activity to simulate the inspiral and merger of binary black holes
three-dimensional general relativistic evolution codes. Several groups
across the world are dedicated to this endeavor but it still lies beyond
present capability. The reasons for the difficulty of the binary black hole
problem reflect the complexity of the underlying physics: the computational
domain has a geometry whose metric is highly dynamic on vastly different time
scales in the inner and outer regions, and it has a topology which is subject
to change in order to avoid singularities. Even in the absence of black holes,
there is no consensus on the best analytic and geometric formalism for dealing
with the nonlinear nature of the gauge freedom and constraints of general
relativity, and a numerical treatment greatly compounds the possibilities. The
problem poses an enormous scientific challenge. See recent reviews
such as [Baumgarte and Shapiro(2003),Shinkai and Yoneda(2002),Lehner(2001)].
Substantial progress on small pieces of the problem has been made by individual
groups but that progress is difficult to assess on a community level for the
purpose of sharing ideas. A newcomer to the field would have to rely completely
on anecdotal evidence in deciding how to develop a code. An observer or
phenomenologist cannot judge the status of a code by wading through pages of
Fortran and needs to view some standard tests to assess its reliability.
Results are often published by various groups that report improvements in
stability or accuracy in evolutions; but formulations, gauges, numerical
methods, boundary conditions and initial data can be inextricably mixed, and it
is often difficult to sort out which ingredients in an improved treatment are
crucial and against what standard the improvement is measured. Different
groups invariably use different criteria for nearly all these issues. This
is a first step toward remedying this situation by the establishment of
standard testbeds that will help the numerical relativity community present and
share results in an objective and useful way that will further scientific
Apart from the practical aspects of providing a collection of standard tests as
a community resource, the formulation of an appropriate test suite presents a
significant scientific challenge. Comparing codes based upon different sets of
variables with different sets of constraint equations is a nontrivial task. It
is not straightforward to draw conclusions which would extend beyond the
particular simulation for which the comparison was made. It is important to
remain aware that evolution systems are a unity of evolution equations,
boundary conditions and gauge. In order to paint a picture which gives
heuristic insight into a broad panorama of ``successes'' and ``failures'', it
is essential to carefully design a set of tests involving different spacetimes
and gauges. The current site is a pivotal first step in this direction.
Although it does not present a complete test suite, it provides some foundation
for building one. We envisage the tests proposed here as the first round in a
series of increasingly more complex tests.
In order to facilitate comparison of results, we will specify all tests
as explicitly as possible. In addition to describing initial data and
gauge, we will also specify a minimal set of output quantities,
the setup of numerical grids, the choice of resolutions for convergence
testing and even numerical methods.
Although our choices aim at broad applicability, it is clear that they will
not be optimal for many formulations. We foresee
the usefulness of additional results, obtained with improved numerical
methods or other modifications in the setup, which may promote further insight
We stress however, that results produced in a simple and identical numerical
setup form the essential basis for a numerical comparison of different
formulations of the equations. We present sample test results which illustrate
and motivate such specifications.
In this initial round of proposed tests, we also keep the spatial resolution
rather low so as not to strain the computational resources of any group.
We believe that this project
introduces a natural framework for documenting algorithms that ``do not work'',
which so far has been sorely lacking but is important for avoiding redundant
experimentation. While failure in one particular case might be easy to detect,
the general task of exploring and comparing different codes over a wide set of
situations is complex.
Already nontrivial is the selection of criteria for comparison. Some standard
- Stability: Exponential growth
(unless on a time scale significantly larger than the relevant physical
timescale) is usually not tolerable in numerical simulations and should not
occur if not inherent in the analytic problem.
- Accuracy: Apart from
resolution and numerical methods (which may be restricted by available
computational resources), accuracy also depends on the analytic formulation.
- Robustness: From a practical point of view, one is interested in a
variety of physical situations. A robust code should be able to perform well in
a wide class of spacetimes, gauges etc.
- Efficiency: For 3D
numerical relativity, time and memory constraints are severe. Thus
computational efficiency is a serious issue.
- Degree of mathematical
understanding: Optimally one would like to mathematically demonstrate certain
features of evolution systems, such as well-posedness or von Neumann stability.
In this work we focus on the issue of stability, and to some degree on
accuracy. By devising a broad set of tests, we also shed some light on the
question of robustness. At this stage, we do not investigate efficiency,
i.e. we do not consider our tests as benchmarks of computational performance
regarding speed or memory requirements. Also, we adopt a practical, empirical
point of view of what works and what does not, regardless of whether a
mathematical theory is available. We hope, of course, that some of our results
might stimulate theoretical progress. Indeed we hope that our work is not only
useful for numerical relativists but also for a more general audience of
relativists who are interested in the current state of numerical simulations.
Furthermore, this work could be useful in facilitating communication between
numerical relativists and the broad community of computational mathematicians
The ideas presented here are an outgrowth of the ``Apples with Apples''
numerical relativity workshop held at UNAM in Mexico City during May 2002. The
objective of the workshop was to formulate some basic tests that could be
carried out by any group with a 3-dimensional code. What we report here are
details of a selection of the first tests that were proposed and the principles
that went into their design. The test results will be published at a later
time. We encourage all groups, whether they attended the workshop or not, to
contribute test results and share authorship in this second publication. While
some tests may not be easily implemented in all formalisms, it is important for
the calibration and support of progress in the field that the various groups
submit as many test results as practical and conform as closely as possible to
the test specifications. A web site www.ApplesWithApples.org has been set up to
coordinate efforts and display results. Specific information about submitting
tests results, and accessing results from other groups, can be obtained there.
A continued series of workshops is being planned with the goal of developing
increasingly more demanding tests leading up to black hole spacetimes with grid
boundaries. The web site provides a forum for proposing new tests and
coordinating plans for future workshops.
The round of preliminary tests presented here is of a simple nature designed to
facilitate broad community participation. All tests use periodic boundary
conditions, which is equivalent to evolution on the 3-torus in the absence of
boundaries. Because the treatment of grid boundaries is one of the most serious
open problems in numerical relativity, it is extremely useful to look in detail
at this case which is not obscured by boundary effects. Only the Gowdy wave
test is based on genuinely nonlinear data. The other tests involve weak and
moderately nonlinear fields. Successful performance of an evolution code at
this level is not obscured by either strong field effects or boundary effects
and is indeed imperative for the successful simulation of black holes. The
tests can be readily performed by any group with the capability of conducting
fully 3-dimensional simulations. The value of this project depends critically
on ease of implementation and flexibility of expansion to future tests.
The main purpose of the tests is to provide a framework for assessing the
accuracy and long term stability of simulations based upon the wide variety of
formulations and numerical approaches being pursued. All present codes are in a
state of flux. The community needs clear information about ``what works and
what doesn't'' in order to carry out the continuous stream of modifications
that must be made. This framework can be used to help compare the different
ideas in meaningful ways; without standard points of reference it is nearly
impossible to assess the effectiveness of one given approach relative to
As might be expected of a rapidly growing field, there is no established
procedure for coordinating code tests with code development. In Section 2, we
provide an overview of some of the current methodology being practiced to
measure accuracy and stability. This provides the background for the discussion
in Section 3 of our strategy in designing a series of code tests and
comparisons which isolate in an effective way a set of performance levels
necessary for successful simulation of black holes. In Section 4, we discuss
the design specifications of the first round of tests, constructed so that code
performance can be based on common output obtained from common input run on
common grids. In Section 5 we summarize our work and discuss future
perspectives. Our goal is to raise issues, collect ideas, point out pitfalls,
document experiences and in general promote and stimulate work toward a better
understanding of what works, what doesn't and - ultimately - why.
We use the following conventions here: the spacetime metric has
signature ; we use geometric units ;
the Ricci tensor is
extrinsic curvature is defined as
where is the induced 3-metric and is the future directed timelike unit
normal; this means that positive extrinsic curvature signifies collapse, and
negative extrinsic curvature expansion.
- Baumgarte and Shapiro(2003)
T. W. Baumgarte
and S. L.
Shapiro, Phys. Rep.
376, 41 (2003),
- Shinkai and Yoneda(2002)
H. Shinkai and
(2002), to be published in Progress in
Astronomy and Astrophysics (Nova Science Publ., New York, 2003),
Class. Quantum Grav. 18,
R25 (2001), gr-qc/0106072.