Subject: COMMS1 Benchmark

From: [email-masked] (Rolf Hempel)
Date: Tue, 13 Jan 1998 14:18:30 +0100

Dear Roger,

thank you for your note on the COMMS1 benchmark. We didn't try the
NEW COMMS1 code yet with our MPI library, so I cannot comment on its
accuracy. I just would like to answer some of the issues you raised
in your mail.

Of course we have seen that in COMMS1 you can select a transition point
between a short and a long model. For this choice, however, you have
to be able to change the input data. In our case (a benchmark suite
used in a procurement) our customer had provided the input dataset,
and we were not allowed to change it. So, the only way for us to correct
the results was to tune our MPI library to make it fit to the benchmark
program. I don't think that this is what you had in mind when you
wrote COMMS1.

You didn't comment on the inaccuracies we found in the raw measurements.
We ran several ping-pong benchmarks before, as, for example, the
MPPTEST routine of MPICH, and they consistently give better latencies
for short messages (difference approx. 25%). As I explained in my
previous mail, we found the reason to be an improper correction for
measurement overheads in COMMS1. Thus, the raw data are flawed,
and this cannot be resolved by any parameter fitting. This is also the
reason that I hesitate to send you the raw data reported by COMMS1 on
our machine.

I agree with you that it would be nice to have a few parameters to
characterize the performance of any given system. The values for
"n1/2" and "rinfinity" have been quite successful for vector arithmetic
operations. The situation is, however, much more complicated for
communication operations.

As an example, let's take the famous ping-pong benchmark. We already
discussed the problem of discontinuities caused by protocol changes.
If you want to do a parameter fitting, the only reasonable solution
seems to me that your test program automatically detects such points
and handles the different protocols separately. If you leave the
selection to an input parameter, you will inevitably run into the
problem I discussed above.

Even if you solve this problem, there remain many others. In modern
(i.e. highly optimized) MPI implementations, the performance of a
ping-pong operation crucially depends on the status of the two processes
involved. Is the receiving process already waiting for the message?
In a ping-pong, it usually is. This can make a huge difference!
Also, the performance can also depend on the global number of processes
active in the application. Not only do search lists in communication
progress engines become shorter if there are fewer processes, but some
implementers even went as far as writing special code for the case
where you just have two processes. Ping-pong codes such as COMMS1
almost always just use two communicating processes, so they measure the
best case. Another effect which is too often ignored is that messages
can interfere with each other (both at the hardware and software level)
if they are sent at the same time between different process pairs.
All those effects combined cause a substantial difference between
ping-pong results and measurements in real applications. In this
situation the apparent precision of performance parameters can be
quite misleading.

If I want to judge the quality of an MPI implementation, I don't
trust in best fit parameters so much. For the ping-pong code, I just
look at a graphic representation of time versus message length for
short messages, and another one of bandwidth versus message length for
long messages. This way I can study discontinuities and other minor
effects in detail. And then, take real applications and measure the
communication times there. Then you will often find surprising 
results which you have never seen in a ping-pong benchmark.

Best wishes,
  Rolf
------------------------------------------------------------------------
Rolf Hempel      (email: [email-masked])
Senior Research Staff Member
C&C Research Laboratories, NEC Europe Ltd., Rathausallee 10,
53757 Sankt Augustin, Germany
Tel.: +49 (0) 2241 - 92 52 - 95
Fax:  +49 (0) 2241 - 92 52 - 99