Subject: Low Level Benchmarks

From: Roger Hockney <[email-masked]>
Date: Sun, 18 Jan 1998 12:24:20 +0000

To: the low-level discussion group

From: Roger

I comment below on recent emailings on this topic which arrived  on the 
16 Jan 1998.

Pat Worley writes:

>2) It may be time to revisit the goals of the Low Level suite. There ar
>   are two obvious extremes.
>
>   a) Determine some (hopefully representative) metrics of point-to-po
>      point communication performance, concentrating on making the    
>      measurements
>      SNIP
>      In this situation, a two (or more) parameter model fit to the 
>      data can be useful, if only as a shorthand for the raw data, 
>      but the model should not be expected to explain the data.

This is of course what COMMS1 sets out to do. But please when judging 
this point, use the New COMMS1 revised code that DOES give much more 
sensible answers in difficult cases. Please do not base your opinions on 
results from the Original COMMS1 code that is still unfortunately being 
issued by Parkbench. Instructions for getting the new code was given in 
my email to this group on 12 Jan 1998. 

>     (The two parameter models are very accurate for some of the 
>      previous generation of homogeneous message-passing platforms.)

It is nice to have confirmation of this from an independent source.
In addition, the 3-parameter mode is available in New COMMS1 for cases 
where the 2-para fails.

>    In case my sympathies are not clear, I prefer to revisit and fix 
>    the current suite, "dumbing it down", if only in presentation, 
>    making it clear what it does and does not measure. 
 
Again this was my objective in writting the New COMMS1 as a minimum fix
to the existing Original COMMS1. However I don't think I would call this
"Dumbing Down".

In fact New COMMS1 is a "Smartening UP" of the benchmark because it
provides a 3-parameter fit for those cases for which the 2-para fit
fails. It also reports the Key spot values of "time for shortest message
(which Charles and Rolfe want to call the Latency)" and bandwidth for
longest message (this could equally well be the maximum measured
bandwidth). It also compares the fitted values with measured values at
these key points. The fit formulae are also given in the output for
completeness.

Pleas note that COMMS1 has always reported ALL the measured lengths and
times in the output file as the basic data, and ALL spot bandwidths were
printed to the screen as measured, and could be captured in a file if
required. In New COMMS1 the spot bandwidths are more conveniently
included in the standard output file as they should have been in the
first place.

Unfortunately the above additions make the new output file more complex
(which I am not happy about). An example of New COMMS1 output is
attached at the end of this email. 

>PPS. - Of course the real reason for using ping-pong is the difficulty 
>       in measuring the time for one-way messaging. I was not aware  
>       that this was a solved problem, at least at the MPI or PVM 
>       level. Perhaps system instrumentation can answer it, but I 
>       didn't know that portable measurement codes could be guaranteed
>       to do so across the different platforms.

Exactly so.

*******************************

Rolf Hempel writes:

>of message-passing. If Charles knows a way to measure single messages,
>I would like to learn about it.

Me too.

>In most other points I agree with Charles. I'm strongly convinced that
>the COMMS* routines are obsolete and should be replaced with something
>reasonable. In particular, the current routines are far too complicated
>to use, and give completely meaningless results. Therefore, I think one

Please base your judgement on the results from New COMMS1 which has a
much more satisfactory fitting procedure (see the examples in the PICT
tool mentioned below). I believe that the revised program New COMMS1
gives reasonable results and is not obselete.

>README file. What is missing is a precise definition of the underlying
>measuring methodology.

In contrast, the methodology of the COMMS1 curve fitting is given in the
Parkbench Report and in detail in my book "The Science of Computer 
Benchmarking", see: 

              http://www.siam.org/catalog/mcc07/hockney.htm 

>I strongly prefer the output of timing tables (perhaps translated in
>good graphical representations) over crude parametrizations like the
>ones in the COMMS* benchmarks. Those can only frustrate the experts
>and confuse all other people.

You seem to have failed to notice that both the Original COMMS1 and the
New COMMS1 report the timing table as the FIRST part of their output
files.

Further a good graphical representation is available using the database
tool from Southampton and my own PICT tool (see below)

The COMMS1 fitting procedure is not crude. On the contrary it uses
least-squares fitting of a performance model that is quite satisfactory
for a lot of data. In minimising relative rather than absolute error,
New COMMS1 spreads the error in a much more satisfactory way and allows
the fitting to be used over a much longer range of message lengths.
Furthermore where the 2-parameter model is unsuitable, New COMMS1
provides a 3-parameter model which fits the Cray T3E (Charles's data 17
Dec 96) very well. I don't think one can call all this crude.

To see how good the 2 and 3 parameter fits produced by New COMMS1 are to 
recent data, check out the examples on my Parkbench Interactive Curve 
Fitting Tool (PICT) at:

          http://www.minnow.demon.co.uk/pict/source/pict2a.html

For the most part these show that 2-parameters fit the data surprisingly 
well. The parameters are not meaningless and useless, but often a
rather good summary of the measurements.

The 3-parameter fit is described quite fully in my talk to the 11 Sep
1997. I have finally written this up with pretty pictures for the PEMCS
Web Journal. Look at:

         http://hpc-journals.ecs.soton.ac.uk/Workshops/PEMCS/fall-97/
                talks/Roger-Hockney/perfprof1.html

In truth we need to see a lot more data before judging the usefulness
of parametric fitting. That is why I would like to look at your NEC 
results. These need not be the timings from COMMS1, but any pingpong 
measurements that you regard as "good".

Please do not base your opinion on the results produced by the Original 
COMMS1 which is presently in the Parkbench suit. This will only work 
satisfactorily results for message lengths up to about 4*10^4. When used 
outside this range it may produce useless numbers.   

>messages are multiples of 4 or 8 bytes, I support Charles' proposal to
>measure the time for an 8 byte message and call it the latency.

I am STRONGLY opposed to this. Latency is an ambiguous term that has 
different meanings to different people. If we wish to report the time 
for an 8-byte message we should call it what it is, no more no less, eg:

                     t(n=8B) = 45.6 us

To call this latency only leads to confusion and senseless 
misunderstanding and argument.

****************************************************************
EXAMPLE NEW COMMS1 OUTPUT FILE: T3E Results from
Grassl's 17 Dec 1996 email to Parkbench committee
****************************************************************

 =================================================
 ===                                           ===
 ===  GENESIS / ParkBench Parallel Benchmarks  ===
 ===                                           ===
 ===              comms1_mpi                   ===
 ===                                           ===
 =================================================

 Pingpong Benchmark:
 -------------------
 Measures time to send a message between two nodes on a multi-processor
 computer (MPP or network) as a function of the message length.

 It also characterises the time and corresponding bandwidth by both two
 and three performance parameters.

 Original code by Roger Hockney (1986/7), modified by Ian Glendinning
 and Ade Miller (1993/4), and by Roger Hockney and Ron Sercely (1997).

 -----------------------------------------------------------------------
 You are running the VERSION dated:    RWH-12-Mar-1997
 -----------------------------------------------------------------------

 The measurement time requested for each test case was  1.00E+00
seconds.

 No distinction was made between long and short messages.

 Zero length messages were not used in least squares fitting.

 -----------------------------------------------
 (1) PRIMARY MEASUREMENTS (BW=Bandwidth, B=Byte)
 -----------------------------------------------------------------------
           SPOT MEASURED VALUES        |   EVOLVING TWO-PARAMETER FIT
 --------------------------------------|--------------------------------
 POINT LENGTH(n)   TIME(t)   BW(r=n/t) |    rinf      nhalf     RMS rel
          B           s         B/s    |    B/s         B       error %
 *SPOT1*-------------------------------|--------------------------------
    1  8.000E+00  1.260E-05  6.349E+05 | 0.000E+00  0.000E+00  0.000E+00
    2  1.000E+01  1.348E-05  7.418E+05 | 2.273E+06  2.064E+01 -1.255E-06
    3  2.000E+01  1.380E-05  1.449E+06 | 1.237E+07  1.516E+02  2.277E+00
    4  3.000E+01  1.590E-05  1.887E+06 | 7.798E+06  9.157E+01  2.762E+00
    5  4.000E+01  1.561E-05  2.562E+06 | 1.020E+07  1.237E+02  3.267E+00
    6  5.000E+01  1.648E-05  3.034E+06 | 1.115E+07  1.366E+02  3.126E+00
    7  6.000E+01  1.618E-05  3.708E+06 | 1.364E+07  1.711E+02  3.796E+00
    8  7.000E+01  1.773E-05  3.948E+06 | 1.356E+07  1.699E+02  3.552E+00
    9  8.000E+01  1.694E-05  4.723E+06 | 1.562E+07  1.992E+02  4.072E+00
   10  9.000E+01  1.793E-05  5.020E+06 | 1.634E+07  2.095E+02  3.954E+00
   11  1.000E+02  1.802E-05  5.549E+06 | 1.741E+07  2.249E+02  3.983E+00
   12  1.100E+02  1.889E-05  5.823E+06 | 1.776E+07  2.300E+02  3.841E+00
   13  1.200E+02  1.780E-05  6.742E+06 | 1.983E+07  2.607E+02  4.483E+00
   14  1.300E+02  1.917E-05  6.781E+06 | 2.034E+07  2.682E+02  4.368E+00
   15  1.400E+02  1.902E-05  7.361E+06 | 2.131E+07  2.828E+02  4.405E+00
   16  1.500E+02  1.941E-05  7.728E+06 | 2.209E+07  2.946E+02  4.389E+00
   17  1.600E+02  1.896E-05  8.439E+06 | 2.353E+07  3.167E+02  4.644E+00
   18  1.700E+02  2.057E-05  8.264E+06 | 2.362E+07  3.179E+02  4.514E+00
   19  1.800E+02  1.911E-05  9.419E+06 | 2.526E+07  3.434E+02  4.887E+00
   20  1.900E+02  2.125E-05  8.941E+06 | 2.517E+07  3.420E+02  4.765E+00
   21  2.000E+02  1.894E-05  1.056E+07 | 2.730E+07  3.754E+02  5.382E+00
   22  2.100E+02  2.091E-05  1.004E+07 | 2.767E+07  3.812E+02  5.282E+00
   23  2.200E+02  2.011E-05  1.094E+07 | 2.885E+07  3.998E+02  5.393E+00
   24  2.300E+02  2.136E-05  1.077E+07 | 2.915E+07  4.047E+02  5.296E+00
   25  2.400E+02  2.015E-05  1.191E+07 | 3.053E+07  4.268E+02  5.496E+00
   26  2.500E+02  2.228E-05  1.122E+07 | 3.047E+07  4.258E+02  5.390E+00
   27  2.600E+02  2.144E-05  1.213E+07 | 3.110E+07  4.360E+02  5.365E+00
   28  2.700E+02  2.212E-05  1.221E+07 | 3.142E+07  4.412E+02  5.290E+00
   29  2.800E+02  2.111E-05  1.326E+07 | 3.249E+07  4.588E+02  5.417E+00
   30  2.900E+02  2.259E-05  1.284E+07 | 3.272E+07  4.626E+02  5.337E+00
   31  3.000E+02  2.284E-05  1.313E+07 | 3.294E+07  4.663E+02  5.262E+00
   32  4.000E+02  2.256E-05  1.773E+07 | 3.550E+07  5.098E+02  5.818E+00
   33  6.000E+02  2.549E-05  2.354E+07 | 4.022E+07  5.921E+02  6.632E+00
   34  8.000E+02  2.817E-05  2.840E+07 | 4.567E+07  6.883E+02  7.296E+00
   35  1.000E+03  3.253E-05  3.074E+07 | 4.887E+07  7.452E+02  7.451E+00
   36  2.000E+03  4.496E-05  4.448E+07 | 5.553E+07  8.657E+02  8.013E+00
   37  5.000E+03  6.135E-05  8.150E+07 | 7.983E+07  1.312E+03  1.090E+01
   38  1.000E+04  8.579E-05  1.166E+08 | 1.070E+08  1.814E+03  1.284E+01
   39  2.000E+04  1.294E-04  1.546E+08 | 1.339E+08  2.315E+03  1.426E+01
   40  3.000E+04  1.722E-04  1.742E+08 | 1.523E+08  2.659E+03  1.493E+01
   41  4.000E+04  2.161E-04  1.851E+08 | 1.647E+08  2.890E+03  1.524E+01
   42  5.000E+04  2.594E-04  1.928E+08 | 1.735E+08  3.056E+03  1.539E+01
   43  1.000E+05  4.534E-04  2.206E+08 | 1.847E+08  3.266E+03  1.575E+01
   44  2.000E+05  7.784E-04  2.569E+08 | 1.996E+08  3.548E+03  1.648E+01
   45  3.000E+05  1.110E-03  2.703E+08 | 2.123E+08  3.787E+03  1.701E+01
   46  5.000E+05  1.697E-03  2.946E+08 | 2.256E+08  4.039E+03  1.762E+01
   47  1.000E+06  3.276E-03  3.053E+08 | 2.370E+08  4.255E+03  1.806E+01
   48  2.000E+06  6.373E-03  3.138E+08 | 2.468E+08  4.440E+03  1.839E+01
   49  3.000E+06  9.489E-03  3.162E+08 | 2.547E+08  4.590E+03  1.858E+01
   50  5.000E+06  1.569E-02  3.187E+08 | 2.612E+08  4.714E+03  1.870E+01
   51  1.000E+07  3.134E-02  3.191E+08 | 2.666E+08  4.816E+03  1.874E+01
 *SPOT2*----------------------------------------------------------------

                            ------------------------
                            COMMS1: Message Pingpong
                            ------------------------
                                 Result Summary    
                                 --------------    

 -------------------
 (2) KEY SPOT VALUES
 -------------------
                                        ----------------------- 
 *KEY1* Shortest  n =   8.000E+00 B,   |  t =   1.260E-05 s    |  ******
                                       |                       |  ******
 *KEY2* Longest   n =   1.000E+07 B,   |  r =   3.191E+08 B/s  |  ******
                                        ----------------------- 

 -----------------------------------------------------------------------
--
 ------------------------------------------
 (3) BEST TWO-PARAMETER LINEAR-(t vs n) FIT
 ------------------------------------------
 (Minimises sum of squares of relative error at all points being fitted)

 Root Mean Square (RMS) Relative Error in time =    18.74 %
                Maximum Relative Error in time =    43.61 %  at POINT =
1

 This is a fit to ALL points. Even though different expressions are
 given for short and long messages, they are algebraically identical
 and either may be used for any message length in the full range.

 --------------
 Short Messages
 --------------
 Best expressions to use if nhalf > 0 and n <= nhalf = 4.816E+03 B

 Bandwidth fitted to:    r = pi0*n/(1+n/nhalf)

 Time fitted to:         t = t0*(1+n/nhalf)

           --------------------------------------------
 *LIN1*   |  pi0 = 5.536E+04 Hz,   nhalf= 4.816E+03 B  |   ******
          |                                            |   ******
 *LIN2*   |   t0 = 1/pi0 = 1.807E-05 s                 |   ******
           --------------------------------------------

 Spot comparison at POINT = 1, n = 8.000E+00 B

   t(fit) = 1.810E-05 s,  t(measured) = 1.260E-05 s,  relative error in
time =   43.6 %

 -------------
 Long Messages
 -------------
 Best expressions to use if  n > nhalf = 4.816E+03 B, or nhalf=0

 Bandwidth fitted to:     r = rinf/(1+nhalf/n)

 Time fitted to:          t = (n+nhalf)/rinf

           -----------------------------------------------
 *LIN3*   |  rinf = 2.666E+08 B/s,  nhalf = 4.816E+03 B   |   ******
           -----------------------------------------------

 Spot comparison at POINT =  51, n = 1.000E+07 B

   r(fit) = 2.665E+08 B/s,  r(measured) = 3.191E+08 B/s,  relative error
in B/W =  -16.5 %

 -----------------------------------------------------------------------
--
 ---------------------------------------
 (4) BEST 3-PARAMETER VARIABLE-POWER FIT
 ---------------------------------------

 Root Mean Square (RMS) Relative Error in B/W =    6.89 %
                Maximum Relative Error in B/W =  -13.41 %  at POINT = 39

 This fit is to ALL data points

 Bandwidth is fitted to:   rvp = rivp/(1+(navp/n)^gamvp)^(1/gamvp)

 Time is fitted to:        tvp = t0vp*(1+(n/navp)^gamvp)^(1/gamvp)

 where                    t0vp = navp/rivp  and  navp = t0vp*rivp

 When gamvp = 1.0, this form reduces to the linear-time form (3) above,
 navp becomes nhalf, and rivp becomes rinf.

 The three independent parameters are (t0vp is derived):

          -------------------------------------------------------------
 *VPWR1* | rivp = 3.475E+08 B/s, navp = 3.670E+03 B, gamvp = 4.190E-01 |
         |                                                             |
 *VPWR2* | t0vp = navp/rivp =  1.056E-05 s                             |  
          -------------------------------------------------------------
 This function is guaranteed to fit the first and last measured values  
 of time  and bandwidth. It also fits the (interpolated) time and  
 bandwidth at n = navp.

-- 
Roger Hockney.  Checkout my new Web page at URL   http://www.minnow.demon.co.uk
University of   and link to my new book: "The Science of Computer Benchmarking"
Westminster UK  suggestions welcome. Know any fish movies or suitable links?

---

From: [email-masked] (Rolf Hempel)
Date: Tue, 20 Jan 1998 14:19:52 +0100

Dear Charles,

thank you for your note, and for sending me your simple test program.
One thing I like about the program is that it's easy to install and run;
no complicated makefiles, include files and sophisticated driver
software. We had the code running in five minutes.

In many points I agree with Ian Glendinning who already reported about
his tests with your code on the Meiko system. When we ran the test on
our SX-4, however, the results were very similar to ping-pong figures.
With the particular MPI version I used for my measurements, the
classical ping-pong test as implemented in MPPTEST of the MPICH
distribution gives about 4 usec less time in latency and about 4%
higher throughput than your test program. The reason for the increase in
latency as reported by your code is fully explained by the fact that
you forgot to correct for the time spent in the timer routine (see
below). So, we would have no problem with adopting a corrected version
of your code as the basic communication test. However, I think that
this is not the point.

The question we have to answer is what communication pattern we want
to measure with our benchmark code. In my view the ping-pong technique,
with all its problems, is much closer to a typical application than
your program. Of course, the situation "receiver already waiting"
implemented by the ping-pong, is a special case which will not be
found for all messages in an application. In this situation,
the MPI implementation can use a more efficient protocol, which will
lead to a best case measurement of latency and throughput.

I agree with Ian that the rapid succession of messages in one direction
is very untypical. Only a stupid programmer would do it this way in
an application, and not aggregate the messages to a larger one. What
you really measure with this benchmark is how well the MPI library
can deal with this kind of congestion. As you see, our library is not
affected at all by this, but, as Ian reported, the Meiko shows a much
different behaviour. In a sense, you measure a kind of worst case
scenario, as opposed to the best case one in the ping-pong.

One technical detail of your program: You time every send operation
separately, and then sum up the individual times. This requires a quite
accurate clock. I would expect that some machines could run into
trouble with this approach. Also, you don't correct for the time needed
for calling the timer twice for every send/receive. On machines with
highly optimized MPI libraries this is not at all negligible. On our
machine two timer calls require as much time as 25% of a complete
send-receive sequence!

As a summary, your basic communication program does not convince me as
a better alternative to ping-pong programs such as MPPTEST. The only
thing I really like about it is its simplicity.

Best regards,
  Rolf
------------------------------------------------------------------------
Rolf Hempel      (email: [email-masked])
Senior Research Staff Member
C&C Research Laboratories, NEC Europe Ltd., Rathausallee 10,
53757 Sankt Augustin, Germany
Tel.: +49 (0) 2241 - 92 52 - 95
Fax:  +49 (0) 2241 - 92 52 - 99