When we looked for a suitable yardstick of performance to rank the TOP500 supercomputers, the availability of results for a wide range of machines was very important for us, as we do not want to exclude machines just because we do not have benchmark data. Looking on the different benchmarks, it was quite clear that we could only take the LINPACK benchmark. This benchmark was introduced by J. Dongarra more than a decade ago and he has been collecting performance data since then [17]. A detailed description as well as a list of performance results on a wide variety of machines is available in postscript form from netlib. To retrieve a copy send electronic mail to netlib@ornl.gov and by typing the message send performance from benchmark or from any machine on the internet type:
rcp anon@netlib2.cs.utk.edu:benchmark/performance performance.
The [i]LINPACK 100[tex2html_wrap3888]100 LINPACK benchmark is a single application benchmark and nowadays has three different variants. The selected problem is to solve a dense system of linear equations. In the first and original variant a system of order 100 has to be solved, also no modifications of the code are allowed at all.
To [i]TPP cope with increasing capabilities of hard- and software a second variant called `Toward Peak Performance (TPP)' was defined by J. Dongarra. In this case the number of equations is 1000 and optimization of the code is permitted, as long as the required precision of the results is achieved.
In [i]HPC the case of MPPs it turned out that the restriction to a system of order 1000 was still too limited to get a realistic impression of the capabilities of these systems. In a third variant called `High parallel computing (HPC)' you are also allowed to scale the problem size to value [tex2html_wrap3890] where you can get optimal performance [tex2html_wrap3892]. As additional information you also have to determine the problem size for which half of the optimal performance is achieved ([tex2html_wrap3894]). Going to bigger machines you can enlarge the problem size to demonstrate the higher capabilities of the system. The achievable performance is now limited by real constraints of the systems like CPU-performance and memory size. This performance does not reflect the overall performance of a given system, as no single number ever can. It does, however, reflect the performance of a dedicated system for solving a dense system of linear equations. As the reported results in general show quite high efficiencies as shown in Figure 3.1, they can be seen as a first realistic correction to the non-achievable peak performance [tex2html_wrap3896]. Also there are examples of other real application codes exceeding the [tex2html_wrap3898] performances [15]. However compared to other application benchmarks (e.g. the NAS PB) the performances especially of MPP systems are quite high.
[figure420]
Due to these properties of LINPACK HPC and TPP and the big number of systems for which results are available, we have chosen the best of both as the basic measure for the TOP500. If in the future a more realistic metric finds widespread usage so that numbers for all systems in question are available, we may convert to that performance measure. But we do not see this happening in the near future due to the big effort involved in it.
To get our list as complete as possible, for the first two releases we used not only numbers published by J. Dongarra, but also reliable numbers we received by communication with the vendors or by linear interpolations between measured results. However, we never used estimates or extrapolations. We will continue to use reliable benchmark results only. The performances we used are shown in Table 3.3, where we also indicate their sources. This table does not show the present LINPACK numbers which can be found in [17], but the ones of 11 November '93, the release date of our second list [16], on which the present TOP500 (Chapter 2) is based.