Subject: Low Level Benchmarks

From: Charles Grassl <[email-masked]>
Date: Thu, 15 Jan 1998 11:11:39 -0600 (CST)

To:      Parkbench interests
From:    Charles Grassl
Subject: Low Level benchmarks

Date:    15 January, 1998

Mark, thank you for pointing us to the PMB benchmark.  It is well written
and coded, but has some discrepancies and shortcomings.  My comments
lead to suggestions and recommendation regarding low level communication
benchmarks.

First, in program PMB the PingPong tests are twice as fast (in time)
as the corresponding message length tests in the PingPing tests (as run
on a CRAY T3E).  The calculation of the time and bandwidth is incorrect
by a factor of 100% in one of the programs.

This error can be fixed by recording, using and reporting the actual
time, amount of data sent and their ratio.  That is, the time should not
be divided by two in order to correct for a round trip.  This recorded
time is for a round trip message, and is not precisely the time for
two messages.  Half the round trip message passing time, as reported in
the PMB tests, is not the time for a single message and should not be
reported and such.  This same erroneous technique is used in the COMMS1
and COMMS2 two benchmarks.  (Is Parkbench is responsible for propagating
this incorrect methodology.)

In program PMB, the testing procedure performs a "warm up".  This
procedure is a poor testing methodology because is discards important
data.  Testing programs such as this should record all times and calculate
the variance and other statistics in order to perform error analysis.

Program PMB does not measure contention or allow extraction of network
contention data.  Tests "Allreduce" and "Bcast" and several others
stress the inter-PE communication network with multiple messages,
but it is not possible to extract information about the contention from
these tests.  The MPI routines for Allreduce and Bcast have algorithms
which change with respect to number of PEs and message lengths,  Hence,
without detailed information about the specific algorithms used, we cannot
extract information about network performance or further characterize
the inter-PE network.

Basic measurements must be separated from algorithms.  Tests PingPong,
PingPing, Barrier, Xover, Cshift and Exchange are low level.  Tests
Allreduce and Bcast are algorithms.  The algorithms Allreduce and Bcast
need additional (algorithmic) information in order to be described in
terms of the basic level benchmarks.

With respect to low level testing, the round trip exchange of messages,
as per PingPing and PingPong in PMB or COMMS1 and COMMS2, is not
characteristic of the lowest level of communication.  This pattern
is actually rather rare in programming practice.  It is more common
for tasks to send single messages and/or to receive single messages.
In this scheme, messages do not make a round trip and there is not
necessarily caching or other coherency effects.

The single message passing is a distinctly different case from that
of round trip tests.  We should be worried that the round trip testing
might introduce artifacts not characteristic of actual (low level) usage.
We need a better test of basic bandwidth and latency in order to measure
and characterize message passing performance.

Here are suggestions and requirements, in an outline form, for a low
level benchmark design:

    I. Single and double (bidirectional) messages.

       A. Test single messages, not round trips.
         1. The round trip test is an algorithm and a pattern.  As
            such it should not be used as the basic low level test of
            bandwidth.
         2. Use direct measurements where possible (which is nearly
            always).  For experimental design, the simplest method is
            the most desirable and best.
         3. Do not perform least squares fits A PIORI.  We know that
            the various message passing mechanisms are not linear or
            analytic because different mechanisms are used for different
            message sizes.  It is not necessarily known before hand
            where this transition occurs.  Some computer systems have
            more than two regimes and their boundaries are dynamic.
         4. Our discussion of least squares fitting is loosing tract
            of experimental design versus modeling.  For example, the
            least squares parameter for t_0 from COMMS1 is not a better
            estimate of latency than actual measurements (assuming
            that the timer resolution is adequate).  A "better" way to
            measure latency is to perform addition DIRECT measurements,
            repetitions or otherwise, and hence decrease the statistical
            error.  The fitting as used in the COMMS programs SPREADS
            error.  It does not reduce error and hence it is not a
            good technique for measuring such an important parameter
            as latency.

       B. Do not test zero length messages.  Though valid, zero length
          messages are likely to take special paths through library
          routines.  This special case is not particularly interesting or
          important.
          1. In practice, the most common and important message size is 64
             bits (one word).  The time for this message is the starting
             point for bandwidth characterization.

       D. Record all times and use statistics to characterize the message
          passing time.  That is, do not prime or warm up caches
          or buffers.  Timings for unprimed caches and buffers give
          interesting and important bounds.  These timings are also the
          nearest to typical usage.  
          1. Characterize message rates by a minimum, maximum, average
             and standard deviation.

       E. Test inhomogeneity of the communication network.  The basic
          message test should be performed for all pairs of PEs.

   II. Contention.

       A. Measure network contention relative to all PEs sending and/or
          receiving messages.

       B. Do not use high level routines where the algorithm is not known.
          1. With high level algorithms, we cannot deduce which component
             of the timing is attributable to the "operation count"
             and which is attributable to the actual system (hardware)
             performance.

  III. Barrier.

       A. Simple test of barrier time for all numbers of processors.

Additionally, the suite should be easy to use.  C and Fortran programs
for direct measurements of message passing times are short and simple.
These simple tests are of order 100 lines of code and, at least in
Fortran 90, can be written in a portable and reliable manner.

The current Parkbench low level suite does not satisfy the above
requirements.  It is inaccurate, as pointed out by previous letters, and
uses questionable techniques and methodologies.  It is also difficult to
use, witness the proliferation of files, patches, directories, libraries
and the complexity and size of the Makefiles.

This Low Level suite is a burden for those who are expecting a tool to
evaluate and investigate computer performance.  The suite is becoming
a liability for our group.  As such, it should be withdrawn from
distribution.

I offer to write, test and submit a new set of programs which satisfy
most of the above requirements.

Charles Grassl
SGI/Cray Research
Eagan, Minnesota  USA

---

From: Mark Baker <[email-masked]>
Date: Mon, 26 Jan 98 10:14:38 GMT

Charles,

Thanks for your thoughts and experiences with the Pallas PMB codes -
I will  forward them to the authors...  The main points in favour of
the PMB codes are that they are in C and potentially produce results
for a variety of MPI calls... Obviously if the results they produce are
flawed...

Regarding new low-level codes I would be in favour of taking up your
kind offer of writing a set of codes in C/Fortran. I guess the main
problem is getting a concensus with regards methodology and measurements
that are used with these codes.

Maybe we can decide that a number of actions should be undertaken...

1) It seems clear that no one is 100% happy with the current version
   of the low-level codes. So, this implies that they need to be 
   replaced !?

2) If we are going to replace the codes we can go down a couple of routes;
   start from scratch, replace with Roger's new codes or some combination of
   both...

3) I would be happy to see us start from scratch and  create 
   C/Fortran codes where the methodology and design of each can be
   "hammered out" by discussion first and then implemented 
   (and iterated as necessary).

4) Assuming that we want to go down this route, I suggest we make a starting
   point of Charles' "suggestions and requirements for the low level 
   benchmark design" - towards the end of this email. I am happy to
   put these words on the web and update/change them as our dicussions
   evolve...

5) Charles has offered his services to help write/design/test these new codes -
   I'm willing to offer my services in a similar fashion. I'm sure that others
   interested in the low-level codes could contribute something here as well. 

Overall, it seems clear to me that we have enough energy and manpower to
produce a new set low-level codes whose methodology and design is correct
and relevant to todays systems...

I look forward to your comments...

Regards

Mark 

---------------End of Original Message-----------------

-------------------------------------
CSM, University of Portsmouth, Hants, UK
Tel: +44 1705 844285	Fax: +44 1705 844006
E-mail: [email-masked]
Date: 01/26/98 - Time: 10:14:38
URL http://www.sis.port.ac.uk/~mab/
-------------------------------------