Subject: Problems with LOWLEVEL comms benchmarks

From: Ron Sercely <[email-masked]>
Date: Mon, 02 Dec 1996 08:54:48 -0600

-- 
Ron Sercely

HP/CXTC Toolsmith
			INTRODUCTION

I recently submitted results to the committee, and since then CXD has released
an implemenation of MPI that results in substatially better performance on
these benchmarks, (up to a factor of 5x), but also reveals some problems with
the comms benchmarks. 

I am unsure how to proceed, hence this wide distribution.
With the current comms1 and comms2 benchmarks, the results are such non-sense
that we cannot report results, so a prompt resolution is necessary.

			MY CONCLUSIONS

Vendors should report startup for short messages, maximum bandwidth and the
message size that produced it, and the result file, only.
A plot of the result file should allow a user of the benchmark get the relevant
information from the benchmark.

			DISCUSSION

Briefly, these benchmarks assume that message bandwidth increases asymtotically
with increasing message size. This assumption does not hold for the current
HP/Convex Division (CXD) implementation.  Caching effects of the CPU and the
fact that we are a shared memory machine results in a maximum bandwidth at a
transfer size comparable to the cache size, with decreasing bandwidth for
larger messages.  Our curve for comms1 looks something like this:     
                                                                    
    |                                                               
    |                                 *                              
    |                          *          *                          
    |                                       *                        *
    |                                           *       *
    |                                                           
    |                   *
    |
    |     
    |               *
    |     
    |     
    |     
    |         *
    |
    |      
    |     
    |    *
    |     
    | *   
    |________________________________________________________________
		Bandwidth vs message size

This causes several problems.
1. The routine that estimates message passing time, to set the loop count,
returns negative transfer times, which results in the comms code using a loop
count of 1, which gives terrible statistics.
2. When this is corrected, the method used to correct times for the "check"
routine overhead is poor.  It measures the time each time through the loop, but
discards the values except for the last iterations.  For large messages, CXD
spends 1/3 of the loop time in this check routine, so measuring this time as
precisely as possible is critical.

With these changes, CXD reliabily gets results similar to the graph above (for
"unmodified" source). To put a scale to this graph, here is sample result file:

 =================================================
 ===                                           ===
 ===  GENESIS / ParkBench Parallel Benchmarks  ===
 ===                                           ===
 ===              comms1_mpi                   ===
 ===                                           ===
 =================================================

 The measurement time requested for each test case was  1.00E+00 seconds.

 SHORT MESSAGES  <=    576B (B=Byte)

 Zero length messages were used in least squares fitting.

    Case  LENGTH(B)   TIME(sec)   RINF(B/s)    N1/2(B)    %error fit  
      1        0      5.180E-06    .000E+00    .000E+00    .000E+00
      2        1      6.884E-06   5.869E+05   3.040E+00    .000E+00
      3        2      6.996E-06   1.102E+06   5.999E+00   5.364E+00
      4       10      7.956E-06   4.966E+06   3.029E+01   7.576E+00
      5       30      1.171E-05   5.352E+06   3.285E+01   4.627E+00
      6      100      1.287E-05   1.493E+07   1.046E+02   1.072E+01
      7      300      1.651E-05   3.063E+07   2.348E+02   1.059E+01
      8      576      2.214E-05   3.812E+07   3.026E+02   8.044E+00

 LONG MESSAGES  >    576B (B=Byte)

    Case  LENGTH(B)   TIME(sec)   RINF(B/s)    N1/2(B)    %error fit  
      9      577      3.254E-05    .000E+00    .000E+00    .000E+00
     10     1000      3.903E-05   6.510E+07   1.541E+03    .000E+00
     11     2000      4.138E-05   1.813E+08   5.634E+03   4.293E+00
     12     5000      5.271E-05   2.443E+08   7.971E+03   3.263E+00
     13    10000      6.792E-05   2.845E+08   9.577E+03   2.700E+00
     14    20000      1.137E-04   2.509E+08   8.096E+03   2.098E+00
     15    50000      2.340E-04   2.479E+08   7.934E+03   9.549E-01
     16   100000      4.459E-04   2.423E+08   7.539E+03   5.952E-01
     17   300000      1.341E-03   2.296E+08   6.107E+03   5.905E-01
     18  1000000      4.539E-03   2.219E+08   4.384E+03   3.466E-01
     19  3000000      1.567E-02   1.933E+08  -1.136E+04   1.277E+00
     20 10000000      5.234E-02   1.910E+08  -1.521E+04   3.807E-01
     21 30000000      1.671E-01   1.802E+08  -6.731E+04   5.413E-01
     22100000000      5.472E-01   1.825E+08  -3.331E+04   1.906E-01

                      ------------------------
                      COMMS1: Message Pingpong
                      ------------------------
                           Result Summmary    
                           ---------------    

 Short Messages, <=  576 Byte
 rinf =    38.116 MByte/s,  nhalf =    302.592 Byte,  startup =     7.939 us

 Long Messages, >  576 Byte
 rinf =   182.491 MByte/s,  nhalf = -33314.086 Byte,  startup =  -182.552 us

			MY CONCLUSIONS

Vendors should report startup for short messages, maximum bandwidth and the
message size that produced it, and the result file, only.

Four changes are necessary to the benchmarks:

1. New ESTCOM routine that does not assume asymtotic behavior.
2. Modified COMMS to correct times based upon total time spent in CHECK.
3. Result Summary should ONLY report startup for short messages, and maximum
bandwidth/message size.  Possible instead of fitting for startup time, the
measured time for 0 or 1 byte messages should be used
4. Results file should print measured, as opposed to fit bandwidth.

I have MPI source to reflect these changes, if the committe would like it.

---

From: Philip Mucci <[email-masked]>
Date: Mon, 2 Dec 1996 14:13:58 -0500

Hi Ron,

If you could prepare us a set of diffs for the code versus the old
code, we could take a look.

Since I started on this project, I have been complaining about the
poor organization of the Low Level benchmarks. In fact, more than
one executive has complained/asked me why they see such poor
performance out of the comms. 

Your patch will definitely do the trick, but in the long term, 
a rewrite will be necessary. In addition, I believe the data 
analysis section should be completely separate from the benchmarks
themselves. The benchmarks should produce data points, and fitting
should be done externally do handle this kind of situation. (I'm sure 
something similar will arise again...)

We should start over and have them rewritten in C, scrapping the output
format and curve fitting code. In addition, they should be written by 
someone who understands end-to-end performance issues. (Remeber, Low Levels
are supposed to tell you what the hardware's capable of, not what application
throughput you can expect.) I'd be happy to help with spec'ing hwo these
benchmarks should be done...

I'm sure the other members will have input on this, we'll see what the
general consensus is. However, I'm sure all the vendor's would like to
start seeing numbers that parallel their own internal measurements...

Send us that patch when you can...

/%*\ Philip J. Mucci | GRA in CS under Dr. JJ Dongarra /*%\
\*%/ http://www.cs.utk.edu/~mucci  PVM/Active Messages \%*/