Subject: NEW COMMS1 benchmark

From: [email-masked] (Rolf Hempel)
Date: Wed, 21 Jan 1998 16:54:00 +0100

In the recent discussion on the low-level benchmarks, Roger repeatedly
asked us to base our evaluation of the COMMS1 benchmark on his new
version, and not on the one which is still in the official PARKBENCH
distribution. At NEC we now have repeated the tests on the NEC SX-4
machine, and I would like to make a few comments on the results.

First of all, the raw data as reported by the table Primary Measurements
more closely match the figures given by other ping-pong tests than the
older version. The correction for oeverheads, however, is still
problematic for the following reasons:
1. In every loop iteration, the returned message is compared with the
   message sent. If one is concerned with the correctnes of the MPI
   library, this could be checked in a separate loop before the timing
   loop. The check inside the timing loop, done only by the sender
   process, delays the sender and thus makes sure that the receiver
   is already waiting in the receive for the next message. This
   aggravates the "Receiver ready" situation which I discussed in an
   earlier mail.
2. The authors take great care in correcting for the overhead introduced
   by the do loop. This is done by the loop over the dummy routine
   before the main loop. On the other hand, the correction for the
   check routine call introduces an overhead of one timer call which
   is NOT taken into account. (Here I assume that the internal clock
   is read out at a fixed point in time during every call of
   DWALLTIME00().) I would argue that on most machines the loop overhead
   per iteration is negligible as compared to a function call. On our
   machine, MPI_Wtime calls a C function which in turn calls an
   assembly language routine. The time needed for this is about 10%
   of our message latency!

Another problem in the measuring procedure is that the test message
contains a single constant, repeated as many times as there are words in
the message. Did the authors never think about the possibility of
data compression in interconnect systems? I would not be surprised to
see bandwidths of Terabytes/sec on some Ethernet connection between
workstations.

Apart from this, the raw data are much better now than they were before,
and when the above points were fixed, the resulting table would be
satisfactory. The interesting question is, however, how much added value
we get from the parameter fitting. In my earlier note, I called the
fitting procedure in the earlier COMMS1 benchmark "crude". I cannot
find a more appropriate word for a model which in cases deviates from
the measured values by more than 100%.

So, how much improvement do we get from the revised COMMS1 version?
As Roger said himself, the increase in modeling sophistication led
to a more complicated output file. Results are now given for two
models, the first one using two parameters, and the second one three.
As could be expected, the two-parameter model does not work better than
in the previous version. For our machine, latency is over-estimated
by 18.9 percent, and the bandwidth at the last data point is off by
27%. Since a linear model is just too simple to be applied to modern
message-passing libraries, I wonder why these results are still in the
output file at all.

The three-parameter fit is better than the two-parameter one. The
major advantage is that it exactly matches the first data point in
time, and the last data point in bandwidth. That is what people would
look at, if there were no parameter fitting at all. So, the reported
latency is the time measured for a zero-byte message, and is as good or
as bad as this measurement. For our MPI library, the RMS fitting
error for the whole data set is 14.04%, and the maximum relative error
is 33.4%. We now can discuss the meaning of the word "crude" (and I
apologize if as a non-native speaker I don't use the right word here),
but I would at least call it unsatisfactory. Given those differences
between model and measurements, I was not surprised to see the
projected RINFINITY as being too high. The 7.65 GBytes/s are well
beyond a memcpy operation in our shared memory, and measured rates never
exceeded 7.1 GBytes/s.

To summarize, in my opinion there is no added value given by the
parameter fitting. The latency value is the first entry in the raw
data table, and the asymptotic bandwidth is easy to figure out by just
looking at the bandwidths as measured for very long messages. As
explained above, the extrapolation by the parametrized model does not
add any precision as compared with a guess based on the long-message
table entries. For message lengths in between, what does a model help
me if it deviates from the measurements by up to 33%? So, my conclusion
would be to drop the whole parameter fitting from the PARKBENCH
low-level routines.

In a separate mail I will send the COMMS1 benchmark output, as produced
with our MPI library, to Roger. I don't want to swamp the whole
PARKBENCH forum with the detailed data.

Best regards,
 Rolf
------------------------------------------------------------------------
Rolf Hempel      (email: [email-masked])
Senior Research Staff Member
C&C Research Laboratories, NEC Europe Ltd., Rathausallee 10,
53757 Sankt Augustin, Germany
Tel.: +49 (0) 2241 - 92 52 - 95
Fax:  +49 (0) 2241 - 92 52 - 99

---

From: Roger Hockney <[email-masked]>
Date: Fri, 23 Jan 1998 16:29:20 +0000

To: The Parkbench discussion group

From: Roger Hockney

First the 3-parameter fit that is produced by New COMMS1 and discussed
by Rolf can be found in the html version of this reply at:

              www.minnow.demon.co.uk/Pbench/emails/hempel1.htm

Or by bringing up the PICT tool on your browser at:

              www.minnow.demon.co.uk/pict/source/pict2a.html
Then:

(1) select a suitable frame size for the PICT display

(2) change the data URL at top from .../data/t3e.res to .../data/sx4.res

(3) press the "GET DATA at URL" button, and the data should download.

(4) press the 3-PARA button then the APPLY3 button, and the 3-para curve
should be drawn.

************************************************************************
Rolf has especialy asked me to point out that the results that he has
supplied are for the SX4 using Release 7.2 MPI software which is will
soon be replaced by a newer version with significantly better latency
and bandwidth. This data does not therefore represent the best that can
be achieved on the SX4.
************************************************************************

I now reply to specific points in Rolf Hempel's email to group on 21 Jan
1998.
 
>machine, and I would like to make a few comments on the results.
Thank you, Rolf, for taking the trouble to install New COMMS1 and
sending me the results. I discuss the results below. In answer to your

>older version. The correction for oeverheads, however, is still

The two points you raise could easily be incorporated in the code. I was
reluctant to tamper with the measurement part of the COMMS1 code because
it would introduce systematic differences in the measurements and make
comparison with older measurements invalid. But of course this has to be
done from time to time. My changes were deliberately kept to a minimum
and confined largely to the parameter fitting part which was causing the
main problems being reported.

>see bandwidths of Terabytes/sec on some Ethernet connection between

Yes I did think about this, but decided I did not know enough about
compression to devise a way to prevent it. Compression algorithms are so
clever now that this may be impossible to do. Anyway this is not yet a
problem, so I suggest we leave it until it becomes one. Perhaps software
should get benefit in its performance numbers for the use of compression
but then we need something more difficult than a sequence of constants
to use as a standard test.
>satisfactory. 

I would have no objection to this.

>The interesting question is, however, how much added value

The added value provided in the case of the NEC SX4 results is that the
3-parameter fit (see graph) gives a satisfactory fit to ALL the data.
This reduces 112 numbers to 3 numbers and an analytic formula that can
be manipulated. This is called "Performance Characterisation" and
provides very useful data compression. Furthermore the parameters
themselves can be interpreted as characterising various aspects of the
shape and asymptotes of the performance curve. 

In contrast reporting just the first time and last performance value and
calling them the Latency and Bandwidth only tells us about these two
points. Further the choice of which message lengths to use for this type
of definition is entirely arbitrary and open to much argument at both
ends. However, New COMMS1 does provide this type of output in the lines
marked KEY SPOT VALUES but I deliberately avoided calling them values of
Latency and Bandwidth in order to avoid senseless argument.

Some people are very interested in the parametric representations,
others not. One is not obliged to use or look at the parametric
representations, but they are there for those who want them. For those
interested just in the Raw data those are reported first in the output
file of New COMMS1.

>message-passing libraries, I wonder why these results are still in the

The 2-PARA results are reported just so that one can see that they are
unsatisfactory, and that therefore one must lose simplicity and consider
a 3-para fit. Actually there is a switch that can be set in the
comms1.inc file to suppress reporting of output if the errors exceed
specified values. Every time I have used this, however, I have tended to
rerun with the output on, in order to see just what the 2-para gave. If
the 2-para can be accepted it is much preferable to the 3-para because
of its simplicity and clearer interpretation of the significance of the
parameters.

>is 33.4%. We now can discuss the meaning of the word "crude" (and I

If you look at the graph itself (see above), I think you will find the
agreement much more satisfactory than is apparent from the reported
errors. You also may have too high an expectation of what parametric
fitting can reasonably be expected to provide, especially for data with
discontinuities. 

In my experience agreement in RMS error rarely is better than 7% and
anything up to 30% is probably still useful. A maximum error of 30% is
not bad at all, and may be due to a single rogue point or an isolated
discontinuity. Although error numbers are reported in the output, one
really has to look at the graph of all data before drawing conclusions. 

>Given those differences

Actually 7.65 differs from 7.1 by 8% which is very good agreement
indeed. 

>looking at the bandwidths as measured for very long messages. As

Your definitions of Latency and Bandwidth will have to be more precise
than the above. What does "by looking at the B/W for very long messages"
actually mean. What are "very long messages?". "What message length
should the first entry in the Raw data table be for?" ... etc.

>table entries. 

Strictly-speaking it is invalid to extrapolate the fitted curve outside
the range of measured values. However we will always do this, and in
this case the fit predicts the known hardware limit as well as can be
reasonably expected.

>For message lengths in between, what does a model help

I think the graph of the results and the 3-para fit shows remarkably
good and useful agreement. But this is a subjective personal opinion.
What do others think?

Best wishes

Roger
-- 
Roger Hockney.  Checkout my new Web page at URL   http://www.minnow.demon.co.uk
University of   and link to my new book: "The Science of Computer Benchmarking"
Westminster UK  suggestions welcome. Know any fish movies or suitable links?