Subject: Low Level benchmarks

From: Charles Grassl <[email-masked]>
Date: Wed, 7 Jan 1998 15:29:59 -0600 (CST)

-- 
Charles Grassl

---

From: Pat Worley <[email-masked]>
Date: Fri, 16 Jan 1998 08:52:17 -0500 (EST)

I have not been paying close attention to the current Low Level communication
suite discussions, having confidence in capabilities and resolve of the
current participants, but have decided to muddy the waters with a few personal
observations. 

1) I do not use the Low Level suite in my own performnace-related work. 
   I find that the interpretation of results is much easier if the
   experiments are designed to answer (my) specific performance questions.
   Producing numbers that are accurate enough and whose experiments are
   well-enough understood to be used to answer arbitrary performance
   questions is much more difficult. 

2) It may be time to revisit the goals of the Low Level suite. There are
   two obvious extremes.

   a) Determine some (hopefully representative) metrics of point-to-point
      communication performance, concentrating on making the measurements
      fair when comparing across platforms, but not requiring that the
      underlying architecture parameters be derivable from these numbers,
      or that they agree exactly with any other group's measurements.
      In this situation, a two (or more) parameter model fit to the data can
      be useful, if only as a shorthand for the raw data, but the model
      should not be expected to explain the data.

   b) Characterize the low level communication performance for each
      platform. Charles Grassl's latest recommendation is a first step in
      that direction. As a personal aside, I attempted such an exercise
      a few years ago (on the T3D, looking at the effect of common usage
      patterns on performance, not just ping-pong between nearest
      neighbors). I quickly became swamped by the amount of data and by the
      number of ways of presenting it (and the work was never written up). I
      realize now that my problem was trying to address too many evaluation
      questions simultaneously. 

      In addition to the large amount of data required, an accurate
      characterization is likely to require more platform-specific
      elements, and will continue to evolve as new machines are added, in
      order to be as fair to the new machines as it is to the old ones.
      (The two parameter models are very acurrate for some of the previous
       generation of homogeneous message-passing platforms.)

    In case my sympathies are not clear, I prefer to revisit and fix the
    current suite, "dumbing it down", if only in presentation, making it clear
    what it does and does not measure. In my own work, the
    point-to-point measurements are only for establishing a general
    performance baseline. The important measures are the performance
    observed in the kernel and full application codes. The baseline
    measurements are simply to assess the "peak achieveable" communication 
    performance.

    While a full characterization is an important thing to do, I do not
    believe that this group has the manpower, resources, or staying power to
    do it right. At one time in the past, we proposed to simply be a
    clearinghouse for the best of the performance measurement codes. If
    Charles wants to write and submit such an extensive low level suite, we
    can consider it, but in the meantime we should address the problems in
    the current suite, and not claim more than is appropriate. In particular,
    make sure that the customer does not become concerned that the
    vendor-stated latency and bandwidth does not match the PARKBENCH reported
    values. A discrepancy does not necessarily mean that someone is lying,
    simply that different aspects are being measured. But we should also be
    sure that intermachine comparisons using PARKBENCH measurements are
    valid, otherwise, they serve no purpose.

Pat Worley

PS.  - I may be in the fringe, but all my codes are written using variants
       of SWAP and SENDRECV, and most of the codes I see can be written in
       such a fashion (and could gain something from it). So, ping-pong and
       ping-ping are not irrelevant to me. 

PPS. - Of course the real reason for using ping-pong is the difficulty in
       measuring the time for one-way messaging. I was not aware that this
       was a solved problem, at least at the MPI or PVM level. Perhaps
       system instrumentation can answer it, but I didn't know that
       portable measurement codes could be guaranteed to do so across the
       different platforms.

---

From: [email-masked] (Rolf Hempel)
Date: Fri, 16 Jan 1998 16:37:14 +0100

I would like to send some remarks to the notes by Charles Grassl and
Pat Worley on the problem of low-level communication benchmarks.

As Pat pointed out, the ping-pong benchmark has been invented because
generally there is no global clock by which you could measure the time
for a single message. Everybody knows that this is no perfect solution,
and in my previous mail I already explained some aspects of why
ping-pong results can differ substantially from times found in real
applications. So, I think we will have to use ping-pong tests in
the future, with the caveat that they only measure a very special case
of message-passing. If Charles knows a way to measure single messages,
I would like to learn about it.

In most other points I agree with Charles. I'm strongly convinced that
the COMMS* routines are obsolete and should be replaced with something
reasonable. In particular, the current routines are far too complicated
to use, and give completely meaningless results. Therefore, I think one
should not even try to correct the COMMS* routines, especially as there
are already better alternatives available. One example is the PMB suite
of PALLAS. It is relatively easy to use, but the documentation should
provide more information than the internal calling tree given in the
README file. What is missing is a precise definition of the underlying
measuring methodology.

I strongly prefer the output of timing tables (perhaps translated in
good graphical representations) over crude parametrizations like the
ones in the COMMS* benchmarks. Those can only frustrate the experts
and confuse all other people.

As to the definition of latency, Charles is right in saying that zero
byte messages are dangerous because they often use special algorithms.
The straightforward solution to use 1 byte messages instead is bad
because usually messages are sent as multiples of 4 or 8 bytes, and for
other message lengths some overhead by additional copying or even
subroutine calls may be introduced. Since the lengths of most real
messages are multiples of 4 or 8 bytes, I support Charles' proposal to
measure the time for an 8 byte message and call it the latency.

I think the warm-up phase before the actual benchmarking is important
in order not to smear out initialization overheads over some number of
messages. The time for the first ping-pong (or other operation), 
however, should be measured and compared with the time found for the
following operations.

I very much welcome Charles Grassl's kind offer to write a new benchmark
suite. Perhaps there are even other suites available which could also
be candidates for getting adopted by PARKBENCH. This forum meanwhile
is quite well-known, which gives them considerable responsibility.
PARKBENCH's choice of benchmark programs influences procurements of new
machines world-wide, and the availability of a good set of low level
benchmarks could give PARKBENCH a good reputation. I'm afraid that the
current set of routines has the opposite effect.

- Rolf Hempel
------------------------------------------------------------------------
Rolf Hempel      (email: [email-masked])
Senior Research Staff Member
C&C Research Laboratories, NEC Europe Ltd., Rathausallee 10,
53757 Sankt Augustin, Germany
Tel.: +49 (0) 2241 - 92 52 - 95
Fax:  +49 (0) 2241 - 92 52 - 99

---

From: Pat Worley <[email-masked]>
Date: Fri, 16 Jan 1998 12:29:01 -0500 (EST)

reasonable. 

I have no problem with this. As I indicated, I have no experience with these.

	What is missing is a precise definition of the underlying measuring
        methodology. 

Perhaps this is the point that I was trying to make. Not only must the
codes be easy to use, but the results should be easy to interpret.
Every code should have a simple description of what it is measuring, what
the data can be used for (and what it shouldn't be used for), and how to use
the data. 

PARKBENCH needs to provide guidance in what data to collect, not just
carefully crafted benchmark codes. And we need to describe clearly what 
low level communication tests are good for. For example,
I have problems with low level contention tests. Understanding hotspots
is an interesting exercise, but the connection to "real" codes
is more subtle. Do we stress test, look at contention for given
algorithms/global operators (and which algorithms), use some standard
workload characterization as the background job, ...? For any given
performance question, what should be used may be clear, but it is difficult
to do this a priori. A simultaneous send/receive stress test may very well be
something interesting to present, but we also need to be able to explain
why (because it is typical in synchronous global communication operations?).

In summary, I would like to see a prioritized list of what low level
information is worth collecting, and why. We can then use this to choose or
generate codes to do the testing. I apologize for being lazy. This may have
already been laid out in the original ParkBench document, but I never worried
about the low level tests before and don't have a copy of the document in
front of me. 

Pat Worley