Subject: Low Level benchmark errors and differences

From: [email-masked] (Rolf Hempel)
Date: Thu, 8 Jan 1998 11:18:48 +0100

To:		Parkbench Low Level interests
From:		Rolf Hempel

Subject:	Low Level benchmark errors and differences,
		Note from Charles Grassl of January 7th

Date:		8 January, 1998

Thank you, Charles, for your note on the Low Level benchmarks. It could
not have come at a better time, because at NEC we just recently ran into
problems with COMMS1.

This code had been specified by a customer as a test case in a current
procurement. When we ran COMMS1 with our current MPI library, the
results for rinfinity and latency were completely wrong. In particular,
the latency values were off by more than a factor of two, when compared
with other ping-pong test programs. The following turned out to be the
main reasons for the errors:

1. The performance model is completely inadequate. A linear dependency
   between time and message length, fitted to the measurements by
   least squares, is bound to fail in the presence of discontinuities
   caused by protocol changes. Most MPI implementations change
   protocols for different message lengths for an overall performance
   optimization.

2. To make things worse, the least square fit overweighs the data points
   for very long messages, because the differences "model minus
   measurement" are largest there in absolute terms. The fitted line,
   therefore, more or less ignores the short message measurements.
   As a result, the latencies are completely up to chance.

3. The correction for internal measurement overhead (e.g., for
   subroutine calls) is programmed in a sloppy way, to say the least.
   We discovered several subroutine calls which were not
   taken into account, and the overhead is measured with low
   precision. For our implementation, this alone introduced a latency
   error of about 25%.
   
The result in our case was that, instead of the 13.5 usec latency
measured by the MPICH MPPTEST routine, COMMS1 initially reported some
28 usec. My colleague Hubert Ritzdorf then made an interesting
experiment: he removed some optimization from our MPI library for
long messages, thus INCREASING the communication times for messages
longer than 128000 bytes, and not changing anything for shorter
messages. The resulting DROP in latency from 28 to under 22 usec
clearly shows how ridiculous the COMMS1 benchmark is.

Thus, I strongly agree with Charles in that the COMMS* benchmarks
must be removed from PARKBENCH. They don't help anybody, and they
only cause confusion on the side of customers and frustration on the
side of benchmarkers. Let's get rid of this long-standing nuisance as
quickly as possible.

Best regards,
 Rolf Hempel
------------------------------------------------------------------------
Rolf Hempel      (email: [email-masked])
Senior Research Staff Member
C&C Research Laboratories, NEC Europe Ltd., Rathausallee 10,
53757 Sankt Augustin, Germany
Tel.: +49 (0) 2241 - 92 52 - 95
Fax:  +49 (0) 2241 - 92 52 - 99

---

From: Roger Hockney <[email-masked]>
Date: Mon, 12 Jan 1998 17:33:01 +0000

To: Rolf, Charles, Mark and others,

From: Roger 

I too am distressed to see the original COMMS1 code (written and tested
for message lengths only up to 10^4) is still being issued by Parkbench
and being used well outside its range of proven validity (message
lengths now typically up to 10^7 or even 10^8).

These problems were pointed out about one year ago by Charles and Ron,
and as a result I worked on the code and issued to the committee a
minmum set of changes to the current release that would solve many of
the problems. These involve replacing five existing routines and adding
two to the existing release. The routines involved have been
downloadable from my Web site since about 12 March 1997 and have been
used successfully at Westminster University in our work. 

The New COMMS1, as I called it, was the subject of two printed reports
to the May 1997 meeting of Parkbench and  further results were shown at
the Sept 1997 meeting. There were also extensive discussions in this
email group during 1997.

Unfortunately my simple fixes were not inserted into the Parkbench
release and as a result we are still getting a bad press from
benchmarkers. After all the effort I put into solving this problem a
year ago, I feel rather let down that my work was never used. If my
changes had been encorporated into the Parkbenchmarks when they were
offered at least as an interim measure, I believe we could have avoided
much of the current bad publicity. 

I emphasise that the New COMMS1 was written as a minimum patch to the
existing release to solve an urgent problem in the simplest way. I am
not against a complete rethink of the low level benchmarks and now that
MPI has become a recognised standard, benchmarks timing the principal
software primitives of MPI would seem to be the most useful. Quite
possibly Charles's or Mucci's codes could be used.

However, I am still firmly convinced of the value of approximate
parametric representation of all the benchmark measurements based on a
simple performance model. Most of the existing low-level benchmarks were
written primarily to determine such parameters and hence include both
raw measurements and least squares curve fitting to obtain the
parameters. I have yet to see data that cannot be satisfactorily fitted
by 2 or 3 parameters, or two sets of 2-paras. And remember that I am
talking here about fitting ALL the measured data by some simple
formulae.    

After the decision of the May 1997 meeting to separate the raw
measurements from the parametric curve fitting, the curve fitting will
eventually become part of the "Parkbench Interactive Curve Fitting Tool"
(PICT). At present this applet can be used to produce a manual curve
fit, but eventually I will put up on my Web site a version in which the
least squares and 3-point buttons are active. But PICT as it is can now
be used manually to see how good or bad the 2-para and 3-para fits are.
Turn your browser to:

       http://www.minnow.demon.co.uk/pict/source/pict2a.html

and insert your raw data. I would be very interested to see what the NEC
data looks like.  

To answer some of Rolf's points: 

Rolf Hempel <[email-masked]> writes
>   protocols for different message lengths for an overall performance
Note that the original COMMS1 that you are using allows you to insert
one break point to take account of one major discontinuity. Have you
tried this?

In any case, to make t_0 a good measure of startup it is sensible ALWAYS
to make a breakpoint at say 100 or 1000 Byte, then the short message t_0
should be a good measure of startup. The long message t_0 is then not of
interest and should be ignored. In this way one is using the straight-
line fit over a short range of lengths, and the resulting t_0 should be
a better estimate of latency because it is derived from several
measurements rather than just selecting a single measurement (e.g. the
time for the shortest message) -- surely a better experimental
procedure. I emphasise that this procedure can be used now with the
original COMMS1 to get sensible results.

If there are many small discontinuities or changes of protocol then I
expect you data is rather like that shown by Charles this time last year
and used as an example in PICT. In this case the 3-para fit may give
good results for your data as it did for Charles's.

>   As a result, the latencies are completely up to chance.
This is absolutely true and was discovered to be the problem one year
ago. My solution, used in the New COMMS1, was and is to minimise the sum
of the squares of the relative (rather than absolute) error. If this is
done the values for short messages are not ignored in the way described,
and t_0 is held much closer to the time for the smallest message length.

Note also that the 3-parameter fit provided by New COMMS1 can be fitted
exactly to the time for the shortest message, to the bandwidth for the
longest message, and to the bandwidth near the mid point. This is the
so-called 3-point fit, but it does require a third parameter.

Can you please email me the output file for the NEC from the original
COMMS1. I can then put this data through the New COMMS1 and see what two
and three parameter fits are produced.

Otherwise you could update your version of Parkbenchmarks with the 7
subroutines and rerun using New COMMS1. See the instructions at the end
of this email.
 
>clearly shows how ridiculous the COMMS1 benchmark is.
Hubert's results are just what one would expect from minimising the
absolute error. I suspect you would not see this effect with New COMMS1
which does not over-emphasise the long message measurements.

Please remember that the t_0 reported by COMMS1 is not a measurement of
the time for any particular message length. It is the constant term in
the fitted curve:

                        t = t_0 + n/rinf

which is an approximation to ALL the measured data.

If you want to know the time, say for the smallest message length, then
that is listed in the table of lengths and times reported in the
benchmark output. If you mean by latency the time for the shortest
message (hopefully zero or 1 Byte) then the COMMS1 measurements of this
are in this table not in t_0.

For those who missed my two earlier emailings on using the New COMMS1, I
copy my earlier email below:

Agenda Item : Plans for the next Release.
              --------------------------
Just a reminder that New COMMS1 as announced in my email to the 
committee of 16 Feb 1997, was designed as the minimum necessary 
changes to the existing release to solve the problems raised at
the beginning of the year. It involves new versions of 5 routines
and 2 new routines. In addition, the Make files need the 2 new 
routines added where appropriate. We have incorporated these 
changes at Westminster in the existing release without trouble.

I believe that these should be incorported in the next release.

In summary:

New COMMS1

In directory:

http://www.minnow.demon.co.uk/Pbench/comms1/

The 5 Changed Routines:

(1) File COMMS1_1.F replaces the following file in the current release:

        ParkBench/Low_Level/comms1/src_mpi/COMMS1.f

(2) File COMMS1_1.INC replaces

        ParkBench/Low_Level/comms1/src_mpi/comms1.inc

(3) File ESTCOM_1.F replaces

        ParkBench/Low_Level/comms1/src_mpi/ESTCOM.f

(4) File LSTSQ_1.F replaces

        ParkBench/lib/Low_Level/LSTSQ.f

(5) File CHECK_1.F replaces

        Parkbench/lib/Low_Level/CHECK.f

The 2 New Routines:

(6) File LINERR_1.F add as

        ParkBench/lib/Low_Level/LINERR.f

(7) File VPOWER_1.F add as

        ParkBench/lib/Low_Level/VPOWER.f

Best wishes to you all

Roger
-- 
Roger Hockney.  Checkout my new Web page at URL   http://www.minnow.demon.co.uk
University of   and link to my new book: "The Science of Computer Benchmarking"
Westminster UK  suggestions welcome. Know any fish movies or suitable links?