Subject: COMMS1

From: Roger Hockney <[email-masked]>
Date: Thu, 9 Jan 1997 16:46:54 +0000

To:      Charles Grassl and the Parkbench Committee
From:    Roger Hockney
Subject: COMMS1 benchmark
Date:    7 January 1997

A happy New Year to you all.

Thank you, Charles, for sending us the complete timing data from the
Cray T3E for the present COMMS1 (PB2.1) and your Proposed New Code 
(PNC). 

After examining this data I am able now to reply to Charles Grassl's
e-mails of 16 and 17 December regarding the COMMS1 benchmark.

As Charles says something needs to be done, so I propose for

IMMEDIATE ACTION:

My conclusion first is that the problems with COMMS1 can be addressed
by relatively minor changes to the existing code. These involve
reporting more prominently the KEY SPOT values already measured
by COMMS1, comparing them with the parametric fit values, and 
changing the fitting procedure. In particular:

  (1) COMMS1.f  Add output of spot bandwidth and Key-spot-values.
                Also add a revison history. I hope Ian Gledinning
                and Mark Papiani will watch what I propose here,
                bearing in mind its effect on automatic processing 
                scripts which are to produce GBIS files from COMMS1
                output files. Ron Sercely, can you please send me 
                your suggested mods to this routine. 
                 
  (2) LSTSQ.f   Add hidden switch to minimising either relative or 
                absolute error. Set switch in program to relative.

  (3) ESTCOM.f  Base estimate of parameters on two spot values
                taken for 1B and MAXLEN Byte. I hope Ian Glendinning
                and Ron Serceley will watch carefully what I do here.
                Ron Sercely, can you please send me your suggested 
                mods to this routine and any others you found it
                necessary to change. 

As the original author of COMMS1 and chairman of the low-level 
Parkbench subcommittee (if this still exists?), I will take on the 
task of submitting revised subroutines ASAP to the group for 
evaluation. I have set a personal deadline of 15 Feb 1997 for this.
I will post proposed code revisions to the group ASAP in the hope 
that some members, particularly Charles and Ron, may have time to 
try them out, and give me some feed back.

Leaving one month for evaluation, this means that revised code
should be available for inclusion in a future release by 15 March
1997. How does this fit into the UTK Parkbench group's schedule
for future releases?  

                     Roger Hockney
                (Westminster University)

For those interested in the details, read on:
You might find it easier to read from my web page which also 
includes various graphs. The URL is

      http://www.minnow.demon.co.uk/Pbench/emails/grassl4.htm

---------------------------------------------------------------

DETAILED RESPONSE TO GRASSL'S EMAILS:

16 December:

(1) PARAMETRIC FIT
------------------
>The COMMS analysis and curve fitting model is incorrect for message
>passing systems with internal buffered libraries and for cache based
>CPUs.  
>For the CRAY T3E,  the COMMS results have an error of 70% and
>hence are not useful.  
 
This is true for the T3D data provided, but we need much more data
before we can draw general conclusions. I find that the timing data 
you submitted on 17 Dec 1996 cannot be fitted accurately by a straight 
line (or two-parameter fit), even if the fitting procedure is modified 
to minimise the sum of the squares of the relative error (as I propose) 
rather than absolute error (as COMMS1 does now). The absolute error 
fitting procedure is certainly not useful and this fitting method 
should be abandoned. The relative error method gives a marginally
acceptable fit in my opinion to this difficult data and confirms my
opinion that the revied COMMS1 should use this method.  

Figure G4_1 shows Grassl's COMMS1(PB2.1) T3E data as green dots
with the red dashed line (fit12) showing the two parameter
(rinf,nhalf) based on a linear timing line that is produced by the 
existing COMMS1 which minimises the square of the absolute errors.
This, as expected, fits the final values for large n well but 
effectively ignores those for small n. This is the curve that 
corresponds, as reported by COMMS1, to
  
       rinf=319.084 MB/s, nhalf= 13112 B, startup= 41.09 us

These figures have been confirmed to be the correct best fit values
using the absolute error minimisation condition. There is no 
numerical error in the COMMS1 or LSTSQ routines. The values however
cause concern because the output file shows that the time to send 
an 8 B message is 12.6 us which is much less than the 41 us startup 
time. That is to say the fit is hopeless at small message lengths. 
However the agreement between rinf (fitted) and the largest 
bandwidth measured is exact to 3 decimal places. An inspection of
the graph shows that the relative error is very unevenly distributed,
being much worse for short messages. I regard this fit as 
unacceptable and certainly not useful. So minimising absolute error
is again found to lead to unsatisfactory results.

If we adopt my suggestion and minimise instead the square of the 
relative error, we obtain fit11 which is shown as a dashed blue 
line and gives

      rinf=267 MB/s,  nhalf= 4816 B,  startup = 18.06 us 

This fit spreads the relative error evenly over the range and
gives a fit that some might regard as just acceptable in giving
a broad view of the performance for ALL message lengths. 

However the fit at small lengths gives a startup of 18 us compared 
to a measured r(n=8B)=12us (50% error), and the fit at long lengths 
gives rinf=267 MB/s compared to a measured r(n=10^7)= 319 MB/s 
(19% error). 

The error in the fit is worst (around 100 %) for intermediate 
values between n= 10^3 and 10^4. Nevertheless I believe that this 
relative-error fit is the most sensible fit to all the data that 
can be produced with two parameters, and confirms my view that 
the revised COMMS1 code should change to minimising the relative
error. If we want a better fit then a third parameter is required 
(more on this later).
  
Note that it is possible with the linear two-parameter fit to 
choose parameters that give an exact fit at the two ends of the
range by equating the time at say 8B to the startup, and the rate 
at say n=10^7 to the final asymptotic value rinf. nhalf is then
derived by calculation. This procedure would give 

    rinf= r(n=10^7 B),  nhalf=startup*rinf,  startup= t(n=8 B)

This amounts to Charles's proposal and matches exactly the spot
values at the two ends and ignores all data in between. This is
shown in Fig. G4_2. Now the linear approximation over-estimates
the performance for all message lengths.  

(2) SPOT VALUES
---------------
In addition to the fitted parameters discussed above it seems to be
forgotten that the existing COMMS1 produces a table of 20 to 40
(typically) spot measurements for message lengths chosen by the
benchmarker in the COMMS1 input data file. These are intended to
be chosen to give a complete picture of the communication
performance for all mesage lengths.

These spot values computed by COMMS1 for each of the specified 
message lengths are valid. When plotted, I can see no significant 
difference (by which I mean >say 20%) between the times reported 
by PNC and COMMS1, excepting that the existing COMMS1 gives 
consistently slightly better results (i.e. lower time values) than 
PNC. I can see some possible reasons for this in the PNC code 
(see below). From this point of view I would expect manufacturers 
to prefer the values measured by COMMS1.

Because of the similarity of the spot values all subsequent work
has been conducted just on the COMMS1 data.

(3) DEFINITIONS
---------------
>(Note:  I have access to accurate Tstart and
>Rinf values and am therefore able to calculate the actual error.)

Charles, to clarify things can you give a definition of what you 
mean by Tstart and Rinf.

Some confusion is occurring I think between the spot measured 
values and fitted performance parameters due to the use of
similar names for both. I suggest we distinguish them more clearly
in the future.

If Tstart means the measured time for the shortest message used,
say 8 B in the case of the T3E measurements, then I think that
this should be reported for exactly what it is, i.e.:

                     t(n=8 B) = ??.?? us

If Rinf means the measured bandwidth at the longest message used,
then again I suggest that this is reported for what it is, i.e.:

                      r(n=10^7 B) = ???.? MB/s

I don't think that one should use the term Rinf for this, because
rinf is the name given to a performance parameter which is a beast
of a different nature.

The performance parameters, on the other hand, are associated 
with some timing model. In the case of (rinf,nhalf) they are
defined as the best values obtained by fitting a range of measured 
(n,t) values to the linear relation t=(n+nhalf)/rinf or t=t0+n/rinf. 

[Whence t0=nhalf/rinf. t0 is called "startup" in the COMMS1 output, 
but t0 (or t sub zero) is the better name.]

Since t0 and rinf arise from a least squares fitting procedure
they should not be expected to correspond exactly to any spot 
value, but if the linear approximation is good and the data does
not have too much experimental scatter, then we expect

             t0   approx= t(n=shortest)
             rinf approx= r(n=longest)

In these circumstances t(shortest) and r(longest) are good estimates
for t0 and rinf. But this is NOT always the case.

If, for example, the experimental data has a lot of scatter
due to the use of an a poor computer clock or an inadequate repeat
value (NREPT) and the linear model is good, we would not want to use  
T(shortest) and r(longest) as estimates for t0 and rinf and ignore 
all the other measured data. The best experimental procedure is to 
fit a straight line to ALL the experimental data and extract t0 and 
rinf from the intercept and slope of the fitted line.

This is EXACTLY what the existing COMMS1 does. 

(4) THEORY BEHIND COMMS1
------------------------
>Though the theory behind the COMMS benchmarks is elegant and useful 
>for modeling, it is not accurate enough for testing and measurement.

Obviously it depends on how linear the experimental timing is as a 
function of message length. We need to see many more results before
drawing conclusions. That's why I am so keen to have people submit
results to the GBIS data base, so we can see how useful the linear 
fit is. And also why it is important that the GBIS plots also 
include the linear-fit line. 

For example, the results obtained at Southampton for the Meiko CS-2 
and reported on page 59 of "The Science of Computer Benchmarking" 
show very good agreement with the linear model: 

   startup(linear fit)=195.2us, t(n=0)=198.3us, t(n=1B)=200.1us; 

    rinf(linear fit)=9.18 MB/s, r(n=40000B=20*nhalf)=8.7 MB/s. 

Also rinf and nhalf are pretty stable from about n=200 B to n=40000 B.
In this case the linear model IS accurate enough for testing and 
measurement.

Can others comment from their own experience, please, on the 
validity/usefulness or otherwise of the linear model?

(5) SYSTEMATIC ERRORS
---------------------
>The errors in the COMMS benchmarks results are not experimental.
>Rather these errors are systematic because they are related to the
>model and modeling.  For example, we can adjust the fitting range and
>parameters and obtain much different different results for Tstart and
>Rinf.  The "jiggling" the model is a systematic effect, not an
>experimental effect.

I agree that the deviations from the linear model are systematic in 
the case of the T3E data. Both rinf and nhalf, which would remain
roughly constant if the data obeyed the linear model, are both seen 
to increase monotonicaly.

However we should not forget that the linear model does fit some 
results very well, in which case it is a very useful and concise
description of the communication performance. It should not be thrown 
away because it does not work well in all circumstances.

(6) THREE-PARAMETER FIT
-----------------------
I have two quite successful 3-parameter fits, but as yet no rational 
as to why they work. The best definition of a third parameter is a 
matter for consideration by those of us who signed up for membership 
of a "Performance evaluation (or some such name)" subcommittee under 
the chaimanship of Aad van der Steen at a past Parkbench meeting.

(6.1) Variable Power Form

This fits the performance(or bandwidth),r, to

                        rinf
          r = --------------------------------
               [1 + (nhalf/n)^gamma]^(1/gamma)

and time
              [n^gamma + nhalf^gamma]^(1/gamma)
          t = ---------------------------------
                         rinf

The three parameters are (rinf,nhalf,gamma). 
When gamma=1 we retrieve the original straight line fit.

This form gives the best fit that I have so far found (SigmaPlot 
reports an error norm of 0.13) and is shown as fit16 in Fig G4_4 
using the parameters

    rinf=365.8 MB/s,   nhalf=4467 B,  gamma=0.446

The fit which is shown as a solid red line is within a few percent 
of all measured spot. In other words it is an excellent fit. The 
only trouble is that I do not know how to interpret the functional 
form or the new parameter gamma.

Strictly speaking nhalf should  now be called n-sub-1/2^(1/gamma)!

(6.2) Hyperbolic Form

This fits a hyperbola to the final asymptotic line with symmetry
about the n-axis, and

                           rinf
           r = ------------------------------- 
                [(1+nhalf/n)^2 - (a/n)^2]^(1/2)
whence

               [(n+nhalf)^2 - a^2]^(1/2)
           t = -------------------------
                         rinf

When a=0 the existing linear model is obtained.

The following values give the best fit with a reported norm of 0.18.

        rinf = 323 MB/s,  nhalf = 43600 B,  a=43340 B

This is plotted as the red line in Fig. G4_5. The fit is also very
good but marginally worse than that of the Variable Power Form.

(7) REPORTING OF COMMS1 RESULTS
-------------------------------
>An alternative and more accurate method of measuring Tstart and Rinf
>would be to use "spot" measurements, where bandwidth measurements are
>made at a small number of sizes. My experiments indicates that the spot
>measurement techniques are accurate, reproducible and reliable
>predictors of general usage.
>I urge the Parkbench group to withdraw the COMMS benchmarks until they
>are fixed.  They are causing great confusion due to their unreliable
>and inaccurate results. 
>I suggest that the COMMS1 and COMMS2 benchmarks be replaced with spot
>measurement programs, of which I have copies.  These types of programs
>are accurate, reliable and easy to use.

But wait a moment, COMMS1 already does this:

COMMS1 starts by measuring some 20 to 40 SPOT values which are 
listed in columns 2 (=n) and 3 (=t) of the output file. I agree that 
a new column should be inserted between 3 and 4 giving the spot 
bandwidth (r=n/t). As mentioned above the existing COMMS1 and PNC 
both give much the same spot values.

COMMS1 also computes a straight-line fit which can be very useful 
when the (n,t) data is roughly linear, but of course is naturally 
not very useful when the data is significantly non linear as in 
Grassl's T3E results.

I think that most of the confusion mentioned above can be solved by
reporting BOTH the "key spot values" as recommended by Charles, 
followed by the results of the two-parameter fit with a comparison
with the "key spot values". That is to say I see the resolution of
the problem principally in the way in which the COMMS1 results are 
reported.      

-----------------------------------------------------------
Date:  17, December 1996

(8) SHORT AND LONG MESSAGES
---------------------------
>I suggest using operationally useful definitions of "short" and 
>"long" messages as follows:

>SHORT MESSAGE: The shortest message which can be sent.  This might 
                be one byte, one word or one cache line.  Note that 
                zero length messages are not of interest because this
length
                message might be trapped by the programmer or the
                message passing library or the OS.

>LONG MESSAGE:  The longest message which can be sent.  This is
                approximately the size of the memory.   It might be 
                one half the size of memory so that there is room 
                for a message passing buffer.

>"Spot" values for measured sizes would be one [byte,word,cache line]
>for short messages and [1,1/2,1/4]*memsize for long messages.  

I agree that zero-length messages might give spurious results and 
should not be used. After all users are surely not really interested 
in sending messages with no information!

The above selection of message lengths gives no values for message 
lengths between SHORT and LONG (after all long starts at say 25MB if 
memsize=10^8B). What about users with messages of say 1KB, 100KB, 
1MB? I think that a benchmark that does not provide information in 
this intermediate range is inadequate.

I favour the present practice of specifying in the COMMS1 data file
20 to 40 message lengths spaced roughly equally on a logarithmic 
scale from 1B to the maximum possible. This tells the WHOLE story in 
the primary result listing for those who want it, after which "Key 
spot values" are selected, followed by the result of the parametric 
fit.  

(9) UNDERSTANDING WHAT'S GOING ON
---------------------------------
>The transition region between short and long messages is complicated
>and is not static.  It is related to cache states and to OS
>parameters for the message passing library.  For a CRAY T3E, the MPI
>buffer has a default value, but can be overridden by a user set
>environment variable.  The default setting is several Kbytes.  The
>cache state is more dynamic and depends on the orientation and
>relative alignment of data and instruction segments.  There are
>generally transitions at sizes 4, 8 and 12 Kbytes in the data cache.
>A linear model can be used for small messages, say less than 100 bytes,
>and for large messages, say more than 500,000 bytes.  But, for cached
>and buffered library message passing, a linear model cannot be made to
>fit both regions combined.  

I find this very interesting because a better understanding of what 
is actually going on would help a lot in trying to find a suitable
third parameter to characterise results like those from the T3E. 
Can you help, Charles, with time versus length equation that might 
apply?

How does a "message passing systems with internal buffered libraries" 
operate in detail?

The measured T3E results actually show a WORSE performance than would 
be seen if a linear model applied with the given Tstart and final
Rinf (see Fig.4_2). In this sense manufacturers might regard making 
their comms system follow a linear timing model as an IDEAL goal.

(10) CPU CACHE
--------------
The presence of a CPU cache should be of no benefit to performance
if the COMMS1 benchmark operates in the way that is intended. This
is because the data to be received by the Master processor is
supposed to come from the slave. If this data is being picked up
directly from cache then the compiler is defeating the object of
the benchmark. Do you think that the presence of the repeat loop
allows this to happen? It is possible that the details of the loop
kernel need to be changed to ensure this does not happen.

(11) PROPOSED REVISED PINGPONG LOOP
-----------------------------------
I am considering changing the basic pinpong loop to: 
(comments invited)

Master sends array to Slave. Slave receives it, adds a number to each 
element and returns it to Master. This ensures that the array is 
used by the Slave code and the data moved out of system buffers to 
user space as is intended. The Master receives the modified array
subtracts the same number from each element and checks the result.
Again the array is used by the master. 

The overhead is measured by having the Master perform the actions in 
sequence of the Slave then the Master with the comms omitted. By this 
I mean the Master adds the number to each element (the Slave action), 
then subtracts it and checks result (the Master action). This is 
repeated NREPT times and timed outside the repeat loop.

It is an error in my view to have any timer calls within the repeat
loop (and COMMS1 does currently have two which need to be removed). 
This is because the sole purpose of the repeat is to allow 
measurements with poor clocks that (by definition) cannot resolve 
time intervals during one pass of the loop. But if the loop is 
repeated say 10^4 times then they can measure its length. For such 
clocks any time difference taken within a single pass of the loop 
may take place in less than a clock tick and be incorrectly recorded 
as zero time or as one clock tick. Multiplying this time by NREPT 
outside the loop does not give a correct value, it only multiplies 
up the error.    

(12) REPLACEMENT OF COMMS1
--------------------------
>The linear model is useful for representing computer network
>performance, but it is not useful for measuring Tstart and Rinf.  The
>direct measurements are more accurate, reliable and easier to program.
>Because of their simplicity and accuracy, they are more useful for low
>level benchmarks.  I suggest that we replace the COMMS1 and COMMS2
>benchmarks with versions similar to what I include below.

Since COMMS1 already does all the direct measurements that are wanted,
the desired result can be achieved by changing the result format
of COMMS1.

(13) COMMENTS ON GRASSL'S PROPOSED CODE
---------------------------------------
I list below the PNC in full with two corrections. They are identified
between lines of  !!!!!!!!!!!!:

  (1) The IF tests to identify the Master and Slave codes should
      go outside the repeat (NREPT) loop. This may partly account for 
      the worse spot results reported by PNC compared with COMMS1.

  (2) The CHECK routine should be inside the Master repeat loop
      otherwise the compiler/system may NOT transfer the
      communicated data from the hidden MPI buffer to the user's
      array A(). The check routine is present not only to test that
      the right numbers are there but also to be sure they are in
      the array A and available to the user's program. This is done 
      by having the program do something with the array A().

FILE NUMBER 2:  Grassl's Tstart and Rinf measurement program
************************************************************************

      PROGRAM COMMS1

      INCLUDE 'mpif.h'

      parameter ( MEMTOT = 100* 2**20 )  ! Memory size in bytes

      INTEGER I, ierr, my_rank, status(MPI_STATUS_SIZE), numprocs,
ibytes
      REAL*8 A(MEMTOT/8)
      REAL   TN,tstart,nhalf,rinf

      CALL MPI_INIT( ierr )
      CALL MPI_COMM_RANK(MPI_COMM_WORLD, my_rank, ierr)
      CALL MPI_COMM_SIZE(MPI_COMM_WORLD, numprocs, ierr)

      IF ( my_rank.EQ.0 ) THEN
         WRITE(*,100) numprocs
      END IF

      tstart = 9999999.
      rinf = 0.
      do I = 0,22
         ibytes = (2**i)*8
         n = max((2**(18-1))/(2**i),10)

         call pingpong(a,ibytes,n,my_rank,tn)

         tmsec = tn*1.e6
         tstart = min(tstart,tmsec)
         rinf =   max(rinf,ibytes/tmsec)
         if ( my_rank.eq.0 ) write(*,101) ibytes,n,tmsec,ibytes/tmsec
      end do

      IF ( my_rank.EQ.0 )THEN
         nhalf = tstart*rinf
         WRITE(*,990)  tstart,nhalf,rinf
      ENDIF

      CALL MPI_FINALIZE(ierr)

      call exit()

  100 FORMAT(//' Message Pingpong'
     &        /' Number of processors:  ',i8
     &       //' Length [Bytes]   Repetitions ',
     &         ' Time [microsec.]   Bandwidth [Mbyte/s]',
     &        /' ', 68('-'))
  101 FORMAT(i10,5x,i10,8x,f10.3,8x,f10.3)
  990 FORMAT(//'        Result Smmmary    ',/,
     &         '        ---------------    ',//,
     &         '   T start: ',f10.1,' microsec.'/
     &         '   N half:  ',f10.1,' bytes'    /
     &         '   R inf:   ',f10.1,' Mbyte/sec.'//)
      END

      subroutine pingpong(a,ilen,nrept,my_rank,tn)
      INCLUDE 'mpif.h'
      real*8 a(ilen/8), DVAL
      INTEGER I, ie, my_rank, status(MPI_STATUS_SIZE)
      REAL T0, T1, T2, TN
      !timer() = MPI_WTIME()
      timer() = 3.33e-9*rtc()

      DVAL = 2.
      A = DVAL

      T1 = timer()
      DO I = 1,NREPT
         CALL DUMMY(I)
      END DO
      T2 = timer()
      T0 = T2-T1

      CALL MPI_BARRIER( MPI_COMM_WORLD, ie )

      T1 = timer()
      DO I = 1,NREPT
         CALL DUMMY(I)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
The following IF test for the Master should go outside the NREPT loop,
as in COMMS1
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 
         IF ( my_rank.EQ.0 ) THEN
            CALL MPI_SEND(A,ILEN,MPI_BYTE,1,10,MPI_COMM_WORLD,ie)
            CALL MPI_RECV(A,ILEN,MPI_BYTE,1,20,MPI_COMM_WORLD,status,ie)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
The Check routine should go here, as it does in COMMS1
CALL CHECK(A, ILEN/8, DVAL)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
         
         ENDIF

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
The following IF test for the Slave should go outside the NREPT loop,
as in COMMS1
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
         IF ( my_rank.EQ.1 ) THEN
            CALL MPI_RECV(A,ILEN,MPI_BYTE,0,10,MPI_COMM_WORLD,status,ie)
            CALL MPI_SEND(A,ILEN,MPI_BYTE,0,20,MPI_COMM_WORLD,ie)
         ENDIF
      END DO
      T2 = timer()
      TN = (T2 - T1 -  T0 )/(NREPT*2)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
The Check routine should be moved inside the NREPT loop (see above), 
as it is in COMMS1
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
      CALL CHECK(A, ILEN/8, DVAL)

      return
      end

      SUBROUTINE CHECK(WRK, LWRK, DVAL)
      INTEGER LWRK
      REAL*8  WRK(LWRK), DVAL
      INTEGER I,IERR
      IERR=0
      DO 10 I = 1, LWRK
         IF ( WRK(I).NE.DVAL ) IERR=IERR+1
   10 CONTINUE
      IF( IERR.NE.0 )THEN
         WRITE(6,*)'ERROR:  DVAL= ',DVAL,' IERR= ',IERR
         STOP
      ENDIF
      RETURN
      END

      SUBROUTINE DUMMY(I)
      INTEGER I
      RETURN
      END

************************************************************************
-- 
Roger Hockney.  Checkout my new Web page at URL   http://www.minnow.demon.co.uk
University of   and link to my new book: "The Science of Computer Benchmarking"
Westminster UK  suggestions welcome. Know any fish movies or suitable links?

---

From: Ron Sercely <[email-masked]>
Date: Wed, 15 Jan 1997 08:11:42 -0600

I have sent Roger the code he asked for in his post, but have several things to discuss
with the committee as a whole.

I belived his goal of 15 Feb is a good one, as it should allow us to have information
for our next meeting in Knoxville. 

1. Although we have been focused on COMMS1, COMMS2 uses the identical (except for the
name of an include file :-( ) ESTCOM routine, so this needs to be changed as well.
Also I suspect that the COMMS2 fitting routines/printing of spot values needs to be
done.

2. Currently, because of the data output, we plot fitted values of bandwidth. I think
this should be replaced with plots of raw bandwidth.  As a user, one of things I would
want, is the ability to ask, "If I use a message length of X on machine Y, how long
does it take?"  The raw data provides this number, the fitted numbers do not.

3. In looking at the fit plotted in figures G4_1 and G4_2. I find both of these to be
unaccepable.  There is a large systematic error, resulting from a fit between reality
and model.  I strongly prefer NO fit to these fits.

4. The fit for HP/Convex data is just as bad.  I have data or postscript files of our
results for comms1, if anyone would like a copy.  They are also contained in a report,
(Frame document or HTML).  Let me know what you want and in what format.

5. The fit shown in figures G4_4 and G4_5 are acceptable, but before we include these
fits in ParkBench, I believe we must verify that other vendors data is equally well
fit.

My problem with bad fits is the following.  Some people will simply look at the
reported values of t0 and rinf.  If they see a lower value t0 (higher value of rinf)
for a particular machine, they will assume that they will attain lower message latency
for small messages (higher measured bandwidth for large messages) IN THEIR APPLICATION,
which may be false using the fit in Figure 1. 

In a more general case, IMHO, if we provide models of communicaion parameters,
users will use these communication models in modeling application performance. If the
model is not _very_ accurate, the user would be much better served by simply using a
table lookup (spot values).

6. Discussion of Caching and Comms performance
> (10) CPU CACHE

It is my belief that CPU cache effects can have a STRONG impact on comms1 performance.

Briefly, blocking message receives complete as soon as data is safely in a users
buffer, ready for access.  This does NOT necessarily mean that the data is within the
receiving processes cache.  In general, the data will be in cache ONLY after
application code access the data.

Consider the example in proposal(11) on  the Exemplar architecture.
Assume that the processes will exchange X bytes of data, the machines have Y byte
caches, and that the processes receive and send into the same buffer..

Consider the cache states for the case X = 2*Y, after startup effects

Master starts timer		Slave is waiting
Master sends array to slave,
and will	not find any data in
cache. (see ** below)
				Slave does a read/modify/write on each array element, starting 					  at the
beginning of the buffer
Master waits on receive		  Because the sender just sent the data, each read is always
				  a cache miss
	
			         Slave sends to Master. Because the send library starts 					 sending from
the beginning of the buffer, each data access
				 will be a cache miss!

Master receives data		Slave calls receive => waits for next send from Master
Master does read/modify/write,
and will cache miss on each 
location  (** ), leaving the
"top" half of the buffer in
cache,  which is what
caused the send library routine miss,
discussed above.

If the master and slave do their read/modify/write _backwards, i.e., starting at the
max value and working backwards, the "bottom" half of the buffer is left in cache, so
when the send library routine is called, the library routine will find half the data
encached!

The overhead method discussed above could account for this, but only if it first, in a
loop, does the add to each element in the array, then, in a separate loop, does the
subtract.  
If performed in a single loop, (the master did an add then a subtract), the caching
would be totally different then in the case with real communication.  

The _current_ comms1 overhead estimation technique is inadequate in this regard, and is
why HP/Convex reports results (as modified code) with the overhead checks removed.

Notice that this example also indicates that although we can design tests to "move data
out of the system buffers", we cannot create tests to ensure that data is in cache!

Now consider the case where x < y.
With communication in place, the Master and Slave will still miss on each cache access.
BUT, the overhead estimation routine will NOT!  The overhead routine will miss on the
first
read/modify/write (when adding the number), but not on the second (when subtracting the
number).
As a result, the estimation of overhead will be low by almost a factor of two,
resulting in calculating much poorer comm1 performance numbers than are actually
obtained.

This case must be addressed, or, we must eliminated this "overhead" loop calculation in
its entirety.  I believe that eliminating the "overhead" loop is the more appropriate
solution.
Again, I believe there is an important distinction between having data available in the
address space of a receiver (mpi_recv completes), and having the data in the cache of
the user (user accesses the data).  I feel we need to ensure that we do NOT measure
caching effects in the comms routines.

-- 
Ron Sercely

HP/CXTC Toolsmith

---

From: Philip Mucci <[email-masked]>
Date: Wed, 15 Jan 1997 17:49:51 -0500

Ron, I believe an easier and more accurate solution to avoid caching effects
would be to "slide" the send and receive buffer pointers along with the test.
I've seen this technique used before...basically it ensures that capacity
misses occur with every block access to comm data...

Allocate send/recv space (at least cache_size*2) call this pointer x.

The sender would do for message size y
for (i=0;i<y/size(x);i++)
send(x+i,y)

Receiver would also do
recv(x+i,y)

There would be 2 outer loops around this, 1 over the message sizes and
1 to make sure the inner send is executed a constant number of times
for each test.

Now, if we are doing a PING-PONG, allow me to make that case that we
should allow cache effects on the master's recv and the slaves send. 
Else it shouldn't be called PING-PONG, but rather request-response. (I'm
sure there's something in the PB paper about this...)

Either way, the same technique can be used for the response. Except now
we have 2 buffers allocated instead of 1, call them a and b.
So now the master would do:

send(a+i,y)
recv(b+i,y)

and the slave
recv(b+i,y)
send(a+i,y)

I think this is much more accurate than introducing a modify step.

Just my 2 cents,

Hope you all had a nice holiday,

-Philip Mucci

/%*\ Philip J. Mucci | GRA in CS under Dr. JJ Dongarra /*%\
\*%/ http://www.cs.utk.edu/~mucci  PVM/Active Messages \%*/