Subject: Re: Low Level benchmark errors and differences

From: Ian Glendinning <[email-masked]>
Date: Mon, 19 Jan 1998 18:52:48 +0100 (MET)

Dear parkbench-comm subscriber,

I have been following the discussions regarding the low-level ParkBench
benchmarks over the last couple of weeks with intertest, but so far I have
been content to keep my head below the parapet, as most of the things I would
have said have been said by others anyway.  However, there is one thing that I
would like to point out.

On Wed Jan  7 22:56:04 1998, Charles Grassl wrote:

> The Low Level programs are obsolete and need to be replaced.

I agree that the existing code could use some improvement, though most of the
discussion seems to have revolved around the version in the "current release",
which as Roger has pointed out several times is very old, and he has written
an improved version.  Have people tried that version out?

> I have
> written seven simple programs, with MPI and PVM versions, and offer them
> as a replacement for the Low Level suite.

I have tried a version of Charles's "comms1" code that he sent me, on our CS-2
system, and found that it reported approximately half the expected asymptotic
bandwidth, so this code is not without its problems either!  By "expected", I
mean the bandwidth reported by various versions of (the ParkBench version of)
COMMS1 over the years, coded using first PARMACS, then PVM, and more recently
MPI, as a message-passing library.  This value corresponds closely to what one
would expect for the peak performance, given the performance figures for the
underlying hardware.  For an explanation of what I think is happening, please
read on...

On Thu Jan 15 20:20:36 1998, Charles Grassl wrote:

> This recorded
> time is for a round trip message, and is not precisely the time for
> two messages.  Half the round trip message passing time, as reported in
> the PMB tests, is not the time for a single message and should not be
> reported and such.  This same erroneous technique is used in the COMMS1
> and COMMS2 two benchmarks.  (Is Parkbench is responsible for propagating
> this incorrect methodology.)

As Pat Worley and Rolf Hempel pointed out, the ping-pong is used because of
the difficulty in measuring the time for one-way messages, and I believe that
this is illustrated in this instance, as it seems that Charles's attempt to
time one-way messages has caused the unexpectedly low asymptotic bandwidth
measurement...  Charles's code executes a send, and then as fast as possible
executes another one, without any concern as to whether the data has left the
sending processor, or has arrived at the receiving processor, and what I think
is happening is that his code is queuing requests to send, before the previous
messages have left the sending processor, forcing the MPI implementation to
buffer them, at the cost of an extra copy operation, which would not otherwise
have been necessary, thus reducing the effective bandwidth!

> With respect to low level testing, the round trip exchange of messages,
> as per PingPing and PingPong in PMB or COMMS1 and COMMS2, is not
> characteristic of the lowest level of communication.  This pattern
> is actually rather rare in programming practice.  It is more common
> for tasks to send single messages and/or to receive single messages.

It seems to me that it is not very common programming practice to send a
sequence of messages to the same destination in rapid fire, without having
either done some intermediate processing, or waiting to get some response
back.  If you were trying to code efficiently, you would doubtless merge
the messages into one, and send the data all together in one message, if it
was all available already, which it must have been if you were able to
execute the sends so rapidly one after another! 

> The single message passing is a distinctly different case from that
> of round trip tests.  We should be worried that the round trip testing
> might introduce artifacts not characteristic of actual (low level) usage.
> We need a better test of basic bandwidth and latency in order to measure
> and characterize message passing performance.

Well, it seems that in this case, the attempt to measure the single message
passing case has introduced an artifact.  To an extent it depends what you are
trying to measure of course, but it has always been my understanding that the
COMMS1 benchmark was trying to measure the peak performance that you could
reasonably expect to obtain using a portable message-passing library
interface, which, for a good implementation of MPI, ought to come close to the
theoretical hardware limit, which is precisely what the existing COMMS1
ping-pong code does on our system.  I would therefore argue in favour of
retaining the ping-pong technique for obtaining timings.
  Ian
--
Ian Glendinning         European Centre for Parallel Computing at Vienna (VCPC)
[email-masked]   Liechtensteinstr. 22, A-1090 Vienna, Austria
Tel: +43 1 310 939612   WWW: http://www.vcpc.univie.ac.at/~ian/