Subject: COMMS benchmarks

From: Charles Grassl <[email-masked]>
Date: Mon, 16 Dec 1996 15:40:12 -0600 (CST)

To:      Parkbench Group
From:    Charles Grassl
Subject: Errors in and usefulness of COMMS benchmarks
Date:    16, December 1996

The COMMS analysis and curve fitting model is incorrect for message
passing systems with internal buffered libraries and for cache based
CPUs.  For the CRAY T3E,  the COMMS results have an error of 70% and
hence are not useful.  (Note:  I have access to accurate Tstart and
Rinf values and am therefore able to calculate the actual error.)
Though the theory behind the COMMS benchmarks is elegant and useful for
modeling, it is not accurate enough for testing and measurement.

The errors in the COMMS benchmarks results are not experimental.
Rather these errors are systematic because they are related to the
model and modeling.  For example, we can adjust the fitting range and
parameters and obtain much different different results for Tstart and
Rinf.  The "jiggling" the model is a systematic effect, not an
experimental effect.

An alternative and more accurate method of measuring Tstart and Rinf
would be to use "spot" measurements, where bandwidth measurements are
made at a small number of sizes. My experiments indicates that the spot
measurement techniques are accurate, reproducible and reliable
predictors of general usage.

I urge the Parkbench group to withdraw the COMMS benchmarks until they
are fixed.  They are causing great confusion due to their unreliable
and inaccurate results. 

I suggest that the COMMS1 and COMMS2 benchmarks be replaced with spot
measurement programs, of which I have copies.  These types of programs
are accurate, reliable and easy to use.

Charles Grassl
Cray Research
Eagan, Minnesota  USA

---

From: Roger Hockney <[email-masked]>
Date: Tue, 17 Dec 1996 12:07:42 +0000

Charles Grassl <[email-masked]> writes
>hence are not useful.  (Note:  I have access to accurate Tstart and
Before we abandon attempts to model message passing behaviour for all
message lengths and replace it with two spot values, I would like to be
sure that the modification to the fitting procedure that I proposed on
13 Dec does not cure the problem.

Charles, could you please send me the one set of results that you
mention above for the T3E, so that I can try the modified fitting method
on it. By this I mean the output file from the existing version of
COMMS1 which gives such inaccurate results.

Also, Charles, do you have any ideas as to what would be a more
satisfactory timing model to use for the systems with buffered libraries
and CPU caches?  

>made at a small number of sizes. 

What spot values do you propose, and how will these be "kept up to date"
as computers change?

Spot-value measurements only give the message performance for very small
and very large messages. They are inaccurate for all other message
lengths. In order to know what large and small mean, we need to have an
estimate for nhalf and to calculate the error factor (1+nhalf/n). For
this we need to fit the whole range of message lengths. If we can make
the fitting method work, we have a model for all message lengths.

>My experiments indicates that the spot
How in detail do you use the  spot values as a reliable predictor for
general usage at some intermediate message length that is therefore not
a spot value?

My proposal of 13 Dec was to provide BOTH revised model fitting and spot
values. Although more complicated this does satisfy both points of view
and provide a clearly visible check between them. Users can then take
their pick.   
-- 
Roger Hockney.  Checkout my new Web page at URL   http://www.minnow.demon.co.uk
University of   and link to my new book: "The Science of Computer Benchmarking"
Westminster UK  suggestions welcome. Know any fish movies or suitable links?

---

From: Charles Grassl <[email-masked]>
Date: Tue, 17 Dec 1996 16:34:47 -0600 (CST)

To:    Roger Hockney
From:  Charles Grassl
Date:  17, December 1996

File number 1 contains output from the parkbench version of program
COMM1.  Note that the times for the short messages shorter than 100 bytes
are f 12 to 18 microseconds.  The least square fit calculates a Tstart of
41 microseconds.  This calculated value does not agree with the actual
measurement.

File number 2 contains my test program with direct or spot measurements.

File number 3 contains output from my test program with direct or spot
measurements.

I suggest using operationally useful definitions of "short" and "long"
messages as follows:

SHORT MESSAGE:  The shortest message which can be sent.  This might be
		one byte, one word or one cache line.  Note that zero
		length messages are not of interest because this length
		message might be trapped by the programmer or the
		message passing library or the OS.

LONG MESSAGE:   The longest message which can be sent.  This is
		approximately the size of the memory.   It might be one
		half the size of memory so that there is room for a
		message passing buffer.

"Spot" values for measured sizes would be one [byte,word,cache line]
for short messages and [1,1/2,1/4]*memsize for long messages.  

The transition region between short and long messages is complicated
and is not static.  It is related to cache states and to OS
parameters for the message passing library.  For a CRAY T3E, the MPI
buffer has a default value, but can be overridden by a user set
environment variable.  The default setting is several Kbytes.  The
cache state is more dynamic and depends on the orientation and
relative alignment of data and instruction segments.  There are
generally transitions at sizes 4, 8 and 12 Kbytes in the data cache.

A linear model can be used for small messages, say less than 100 bytes,
and for large messages, say more than 500,000 bytes.  But, for cached
and buffered library message passing, a linear model cannot be made to
fit both regions combined.  

The linear model is useful for representing computer network
performance, but it is not useful for measuring Tstart and Rinf.  The
direct measurements are more accurate, reliable and easier to program.
Because of their simplicity and accuracy, they are more useful for low
level benchmarks.  I suggest that we replace the COMMS1 and COMMS2
benchmarks with versions similar to what I include below.

Regards,
Charles Grassl
Cray Research

FILE NUMBER 1:  Output from Parkbench COMMS1 program
************************************************************************
 =================================================
 ===                                           ===
 ===  GENESIS / ParkBench Parallel Benchmarks  ===
 ===                                           ===
 ===              comms1_mpi                   ===
 ===                                           ===
 =================================================
 
 The measurement time requested for each test case was  1.00E+00 seconds

 No distinction was made between long and short messages.

 Zero length messages were not used in least squares fitting.

    Case  LENGTH(B)   TIME(sec)   RINF(B/s)    N1/2(B)    %error fit  
      1        8      1.260E-05   0.000E+00   0.000E+00   0.000E+00
      2       10      1.348E-05   2.277E+06   2.070E+01   0.000E+00
      3       20      1.380E-05   1.293E+07   1.592E+02   2.168E+00
      4       30      1.590E-05   7.656E+06   8.976E+01   2.414E+00
      5       40      1.561E-05   1.032E+07   1.257E+02   3.129E+00
      6       50      1.648E-05   1.140E+07   1.407E+02   2.849E+00
      7       60      1.618E-05   1.420E+07   1.800E+02   3.580E+00
      8       70      1.773E-05   1.396E+07   1.765E+02   3.061E+00
      9       80      1.694E-05   1.632E+07   2.107E+02   3.739E+00
     10       90      1.793E-05   1.712E+07   2.224E+02   3.422E+00
     11      100      1.802E-05   1.834E+07   2.405E+02   3.430E+00
     12      110      1.889E-05   1.864E+07   2.450E+02   3.145E+00
     13      120      1.780E-05   2.104E+07   2.815E+02   3.979E+00
     14      130      1.917E-05   2.155E+07   2.893E+02   3.589E+00
     15      140      1.902E-05   2.268E+07   3.069E+02   3.651E+00
     16      150      1.941E-05   2.357E+07   3.208E+02   3.563E+00
     17      160      1.896E-05   2.529E+07   3.479E+02   3.894E+00
     18      170      2.057E-05   2.519E+07   3.463E+02   3.490E+00
     19      180      1.911E-05   2.716E+07   3.779E+02   4.129E+00
     20      190      2.125E-05   2.680E+07   3.720E+02   3.633E+00
     21      200      1.894E-05   2.929E+07   4.125E+02   4.702E+00
     22      210      2.091E-05   2.965E+07   4.184E+02   4.174E+00
     23      220      2.011E-05   3.110E+07   4.424E+02   4.457E+00
     24      230      2.136E-05   3.136E+07   4.467E+02   4.113E+00
     25      240      2.015E-05   3.305E+07   4.751E+02   4.564E+00
     26      250      2.228E-05   3.274E+07   4.697E+02   4.057E+00
     27      260      2.144E-05   3.348E+07   4.824E+02   4.196E+00
     28      270      2.212E-05   3.378E+07   4.874E+02   4.004E+00
     29      280      2.111E-05   3.510E+07   5.101E+02   4.326E+00
     30      290      2.259E-05   3.527E+07   5.129E+02   3.979E+00
     31      300      2.284E-05   3.543E+07   5.157E+02   3.874E+00
     32      400      2.256E-05   3.902E+07   5.800E+02   4.479E+00
     33      600      2.549E-05   4.647E+07   7.168E+02   4.641E+00
     34      800      2.817E-05   5.431E+07   8.625E+02   4.602E+00
     35     1000      3.253E-05   5.715E+07   9.160E+02   3.988E+00
     36     2000      4.496E-05   6.571E+07   1.081E+03   3.115E+00
     37     5000      6.135E-05   1.022E+08   1.813E+03   4.116E+00
     38    10000      8.579E-05   1.353E+08   2.503E+03   3.932E+00
     39    20000      1.294E-04   1.683E+08   3.230E+03   3.396E+00
     40    30000      1.722E-04   1.854E+08   3.632E+03   2.882E+00
     41    40000      2.161E-04   1.950E+08   3.871E+03   2.459E+00
     42    50000      2.594E-04   2.012E+08   4.041E+03   2.156E+00
     43   100000      4.534E-04   2.203E+08   4.678E+03   1.748E+00
     44   200000      7.784E-04   2.515E+08   5.978E+03   1.817E+00
     45   300000      1.110E-03   2.668E+08   6.743E+03   1.536E+00
     46   500000      1.697E-03   2.871E+08   8.080E+03   1.445E+00
     47  1000000      3.276E-03   3.022E+08   9.529E+03   9.226E-01
     48  2000000      6.373E-03   3.121E+08   1.097E+04   5.644E-01
     49  3000000      9.489E-03   3.155E+08   1.165E+04   4.020E-01
     50  5000000      1.569E-02   3.180E+08   1.248E+04   2.620E-01
     51 10000000      3.134E-02   3.191E+08   1.311E+04   1.344E-01

                      ------------------------
                      COMMS1: Message Pingpong
                      ------------------------
                           Result Summary    
                           ---------------    

 rinf =   319.084 MByte/s,  nhalf =  13112.553 Byte,  startup =    41.09
************************************************************************

FILE NUMBER 2:  Grassl's Tstart and Rinf measurement program
************************************************************************

      PROGRAM COMMS1

      INCLUDE 'mpif.h'

      parameter ( MEMTOT = 100* 2**20 )  ! Memory size in bytes

      INTEGER I, ierr, my_rank, status(MPI_STATUS_SIZE), numprocs, ibytes
      REAL*8 A(MEMTOT/8)
      REAL   TN,tstart,nhalf,rinf

      CALL MPI_INIT( ierr )
      CALL MPI_COMM_RANK(MPI_COMM_WORLD, my_rank, ierr)
      CALL MPI_COMM_SIZE(MPI_COMM_WORLD, numprocs, ierr)

      IF ( my_rank.EQ.0 ) THEN
         WRITE(*,100) numprocs
      END IF

      tstart = 9999999.
      rinf = 0.
      do I = 0,22
         ibytes = (2**i)*8
         n = max((2**(18-1))/(2**i),10)

         call pingpong(a,ibytes,n,my_rank,tn)

         tmsec = tn*1.e6
         tstart = min(tstart,tmsec)
         rinf =   max(rinf,ibytes/tmsec)
         if ( my_rank.eq.0 ) write(*,101) ibytes,n,tmsec,ibytes/tmsec
      end do

      IF ( my_rank.EQ.0 )THEN
         nhalf = tstart*rinf
         WRITE(*,990)  tstart,nhalf,rinf
      ENDIF

      CALL MPI_FINALIZE(ierr)

      call exit()

  100 FORMAT(//' Message Pingpong'
     &        /' Number of processors:  ',i8
     &       //' Length [Bytes]   Repetitions ',
     &         ' Time [microsec.]   Bandwidth [Mbyte/s]',
     &        /' ', 68('-'))
  101 FORMAT(i10,5x,i10,8x,f10.3,8x,f10.3)
  990 FORMAT(//'        Result Smmmary    ',/,
     &         '        ---------------    ',//,
     &         '   T start: ',f10.1,' microsec.'/
     &         '   N half:  ',f10.1,' bytes'    /
     &         '   R inf:   ',f10.1,' Mbyte/sec.'//)
      END

      subroutine pingpong(a,ilen,nrept,my_rank,tn)
      INCLUDE 'mpif.h'
      real*8 a(ilen/8), DVAL
      INTEGER I, ie, my_rank, status(MPI_STATUS_SIZE)
      REAL T0, T1, T2, TN
      !timer() = MPI_WTIME()
      timer() = 3.33e-9*rtc()

      DVAL = 2.
      A = DVAL

      T1 = timer()
      DO I = 1,NREPT
         CALL DUMMY(I)
      END DO
      T2 = timer()
      T0 = T2-T1

      CALL MPI_BARRIER( MPI_COMM_WORLD, ie )

      T1 = timer()
      DO I = 1,NREPT
         CALL DUMMY(I)
         IF ( my_rank.EQ.0 ) THEN
            CALL MPI_SEND(A,ILEN,MPI_BYTE,1,10,MPI_COMM_WORLD,ie)
            CALL MPI_RECV(A,ILEN,MPI_BYTE,1,20,MPI_COMM_WORLD,status,ie)
         ENDIF
         IF ( my_rank.EQ.1 ) THEN
            CALL MPI_RECV(A,ILEN,MPI_BYTE,0,10,MPI_COMM_WORLD,status,ie)
            CALL MPI_SEND(A,ILEN,MPI_BYTE,0,20,MPI_COMM_WORLD,ie)
         ENDIF
      END DO
      T2 = timer()
      TN = (T2 - T1 -  T0 )/(NREPT*2)

      CALL CHECK(A, ILEN/8, DVAL)

      return
      end

      SUBROUTINE CHECK(WRK, LWRK, DVAL)
      INTEGER LWRK
      REAL*8  WRK(LWRK), DVAL
      INTEGER I,IERR
      IERR=0
      DO 10 I = 1, LWRK
         IF ( WRK(I).NE.DVAL ) IERR=IERR+1
   10 CONTINUE
      IF( IERR.NE.0 )THEN
         WRITE(6,*)'ERROR:  DVAL= ',DVAL,' IERR= ',IERR
         STOP
      ENDIF
      RETURN
      END

      SUBROUTINE DUMMY(I)
      INTEGER I
      RETURN
      END

************************************************************************

FILE NUMBER 3:  Output from Grassl's Tstart and Rinf measurement program
************************************************************************

 Message Pingpong
 Number of processors:         2

 Length [Bytes]   Repetitions  Time [microsec.]   Bandwidth [Mbyte/s]
 --------------------------------------------------------------------
         8         131072            15.153             0.528
        16          65536            15.528             1.030
        32          32768            20.972             1.526
        64          16384            20.854             3.069
       128           8192            22.510             5.686
       256           4096            25.898             9.885
       512           2048            29.601            17.297
      1024           1024            37.772            27.110
      2048            512            51.092            40.085
      4096            256            77.785            52.658
      8192            128            92.190            88.860
     16384             64           130.623           125.429
     32768             32           210.075           155.982
     65536             16           353.894           185.185
    131072             10           596.613           219.693
    262144             10          1017.277           257.692
    524288             10          1849.011           283.551
   1048576             10          3482.081           301.135
   2097152             10          6830.946           307.008
   4194304             10         13538.897           309.797
   8388608             10         26720.909           313.934
  16777216             10         53281.705           314.878
  33554432             10        106447.112           315.222

        Result Summary    
        ---------------    

   T start:       15.2 microsec.
   N half:      4776.5 bytes
   R inf:        315.2 Mbyte/sec.