Introduction

The inspiral of a relativistic binary has played the role of a standard candle for the first signal to be detected by the gravitational wave observatories which are now approaching operational readiness. For many years now, this has spurred activity to simulate the inspiral and merger of binary black holes using fully three-dimensional general relativistic evolution codes. Several groups across the world are dedicated to this endeavor but it still lies beyond present capability. The reasons for the difficulty of the binary black hole problem reflect the complexity of the underlying physics: the computational domain has a geometry whose metric is highly dynamic on vastly different time scales in the inner and outer regions, and it has a topology which is subject to change in order to avoid singularities. Even in the absence of black holes, there is no consensus on the best analytic and geometric formalism for dealing with the nonlinear nature of the gauge freedom and constraints of general relativity, and a numerical treatment greatly compounds the possibilities. The problem poses an enormous scientific challenge. See recent reviews such as [Baumgarte and Shapiro(2003),Shinkai and Yoneda(2002),Lehner(2001)].

Substantial progress on small pieces of the problem has been made by individual groups but that progress is difficult to assess on a community level for the purpose of sharing ideas. A newcomer to the field would have to rely completely on anecdotal evidence in deciding how to develop a code. An observer or phenomenologist cannot judge the status of a code by wading through pages of Fortran and needs to view some standard tests to assess its reliability. Results are often published by various groups that report improvements in stability or accuracy in evolutions; but formulations, gauges, numerical methods, boundary conditions and initial data can be inextricably mixed, and it is often difficult to sort out which ingredients in an improved treatment are crucial and against what standard the improvement is measured. Different groups invariably use different criteria for nearly all these issues. This is a first step toward remedying this situation by the establishment of standard testbeds that will help the numerical relativity community present and share results in an objective and useful way that will further scientific progress.

Apart from the practical aspects of providing a collection of standard tests as a community resource, the formulation of an appropriate test suite presents a significant scientific challenge. Comparing codes based upon different sets of variables with different sets of constraint equations is a nontrivial task. It is not straightforward to draw conclusions which would extend beyond the particular simulation for which the comparison was made. It is important to remain aware that evolution systems are a unity of evolution equations, boundary conditions and gauge. In order to paint a picture which gives heuristic insight into a broad panorama of ``successes'' and ``failures'', it is essential to carefully design a set of tests involving different spacetimes and gauges. The current site is a pivotal first step in this direction. Although it does not present a complete test suite, it provides some foundation for building one. We envisage the tests proposed here as the first round in a series of increasingly more complex tests.

In order to facilitate comparison of results, we will specify all tests as explicitly as possible. In addition to describing initial data and gauge, we will also specify a minimal set of output quantities, the setup of numerical grids, the choice of resolutions for convergence testing and even numerical methods. Although our choices aim at broad applicability, it is clear that they will not be optimal for many formulations. We foresee the usefulness of additional results, obtained with improved numerical methods or other modifications in the setup, which may promote further insight and clarification. We stress however, that results produced in a simple and identical numerical setup form the essential basis for a numerical comparison of different formulations of the equations. We present sample test results which illustrate and motivate such specifications. In this initial round of proposed tests, we also keep the spatial resolution rather low so as not to strain the computational resources of any group.

We believe that this project introduces a natural framework for documenting algorithms that ``do not work'', which so far has been sorely lacking but is important for avoiding redundant experimentation. While failure in one particular case might be easy to detect, the general task of exploring and comparing different codes over a wide set of situations is complex.

Already nontrivial is the selection of criteria for comparison. Some standard criteria are:

1. Stability: Exponential growth (unless on a time scale significantly larger than the relevant physical timescale) is usually not tolerable in numerical simulations and should not occur if not inherent in the analytic problem.
2. Accuracy: Apart from resolution and numerical methods (which may be restricted by available computational resources), accuracy also depends on the analytic formulation.
3. Robustness: From a practical point of view, one is interested in a variety of physical situations. A robust code should be able to perform well in a wide class of spacetimes, gauges etc.
4. Efficiency: For 3D numerical relativity, time and memory constraints are severe. Thus computational efficiency is a serious issue.
5. Degree of mathematical understanding: Optimally one would like to mathematically demonstrate certain features of evolution systems, such as well-posedness or von Neumann stability.

In this work we focus on the issue of stability, and to some degree on accuracy. By devising a broad set of tests, we also shed some light on the question of robustness. At this stage, we do not investigate efficiency, i.e. we do not consider our tests as benchmarks of computational performance regarding speed or memory requirements. Also, we adopt a practical, empirical point of view of what works and what does not, regardless of whether a mathematical theory is available. We hope, of course, that some of our results might stimulate theoretical progress. Indeed we hope that our work is not only useful for numerical relativists but also for a more general audience of relativists who are interested in the current state of numerical simulations. Furthermore, this work could be useful in facilitating communication between numerical relativists and the broad community of computational mathematicians and physicists.

The ideas presented here are an outgrowth of the ``Apples with Apples'' numerical relativity workshop held at UNAM in Mexico City during May 2002. The objective of the workshop was to formulate some basic tests that could be carried out by any group with a 3-dimensional code. What we report here are details of a selection of the first tests that were proposed and the principles that went into their design. The test results will be published at a later time. We encourage all groups, whether they attended the workshop or not, to contribute test results and share authorship in this second publication. While some tests may not be easily implemented in all formalisms, it is important for the calibration and support of progress in the field that the various groups submit as many test results as practical and conform as closely as possible to the test specifications. A web site www.ApplesWithApples.org has been set up to coordinate efforts and display results. Specific information about submitting tests results, and accessing results from other groups, can be obtained there. A continued series of workshops is being planned with the goal of developing increasingly more demanding tests leading up to black hole spacetimes with grid boundaries. The web site provides a forum for proposing new tests and coordinating plans for future workshops.

The round of preliminary tests presented here is of a simple nature designed to facilitate broad community participation. All tests use periodic boundary conditions, which is equivalent to evolution on the 3-torus in the absence of boundaries. Because the treatment of grid boundaries is one of the most serious open problems in numerical relativity, it is extremely useful to look in detail at this case which is not obscured by boundary effects. Only the Gowdy wave test is based on genuinely nonlinear data. The other tests involve weak and moderately nonlinear fields. Successful performance of an evolution code at this level is not obscured by either strong field effects or boundary effects and is indeed imperative for the successful simulation of black holes. The tests can be readily performed by any group with the capability of conducting fully 3-dimensional simulations. The value of this project depends critically on ease of implementation and flexibility of expansion to future tests.

The main purpose of the tests is to provide a framework for assessing the accuracy and long term stability of simulations based upon the wide variety of formulations and numerical approaches being pursued. All present codes are in a state of flux. The community needs clear information about ``what works and what doesn't'' in order to carry out the continuous stream of modifications that must be made. This framework can be used to help compare the different ideas in meaningful ways; without standard points of reference it is nearly impossible to assess the effectiveness of one given approach relative to another.

As might be expected of a rapidly growing field, there is no established procedure for coordinating code tests with code development. In Section 2, we provide an overview of some of the current methodology being practiced to measure accuracy and stability. This provides the background for the discussion in Section 3 of our strategy in designing a series of code tests and comparisons which isolate in an effective way a set of performance levels necessary for successful simulation of black holes. In Section 4, we discuss the design specifications of the first round of tests, constructed so that code performance can be based on common output obtained from common input run on common grids. In Section 5 we summarize our work and discuss future perspectives. Our goal is to raise issues, collect ideas, point out pitfalls, document experiences and in general promote and stimulate work toward a better understanding of what works, what doesn't and - ultimately - why.

We use the following conventions here: the spacetime metric has signature ; we use geometric units ; the Ricci tensor is ; the extrinsic curvature is defined as ; where is the induced 3-metric and is the future directed timelike unit normal; this means that positive extrinsic curvature signifies collapse, and negative extrinsic curvature expansion.

Baumgarte and Shapiro(2003)

T. W. Baumgarte and S. L. Shapiro, Phys. Rep. 376, 41 (2003), gr-qc/0211028.

Shinkai and Yoneda(2002)

H. Shinkai and G. Yoneda (2002), to be published in Progress in Astronomy and Astrophysics (Nova Science Publ., New York, 2003), gr-qc/0209111.

Lehner(2001)

L. Lehner, Class. Quantum Grav. 18, R25 (2001), gr-qc/0106072.