Subject: Low Level benchmark errors and differences

From: Charles Grassl <[email-masked]>
Date: Wed, 07 Jan 1998 15:36:19 -0600

To:      Parkbench Low Level interests
From:    Charles Grassl

Subject: Low Level benchmark errors and differences

Date:    7 January, 1998

We should not produce or publish Parkbench Low level benchmark results
with the current suite of programs because the programs are inaccurate
and unreliable.  I ran the Low Level programs and compared the results
with the same metrics as recorded from other benchmark programs.
The differences range from less than 5% (acceptable) to a factor of 6
times difference, which is unacceptable.

The differences, or "errors", are summarized in the table below.
The recorded differences in results from the Low Level program were
arrived at by comparing the Parkbench program reported metrics with the
same metrics as measured by alternative programs.

       Table.  Differences in Low Level benchmark results
               for two systems.  System A is an Origin 2000.
               System B is a CRAY T3E.

                     System A          System B
                  Rinf  Startup    Rinf   Startup
        -----------------------------------------
        COMMS1    <10%     6x       <5%      6x
        COMMS2      2x     3x       <5%     <5%
        COMMS3     <5%              <5%
        POLY1      <5%    60%        2x     <5%
        POLY2      <5%    60%        2x     <5%
        POLY3       -      -         2x     80x

The Parkbench Low Level programs are occasionally requested for
benchmarking computer systems, but the results are usually rejected
because of their inaccuracy and unreliability.  If not rejected, they
cause confusion and consternation because the results do not agree
with other measurements of the same variables.  I emphasize that this
is not a case of obtaining optimization and favorable results for a
computer system.  The problem is with the inaccuracy and unreliability
of the results.

The Low Level programs measure and report low level parameters.
Therefore their value is in accuracy and utility.  The programs do not
constitute definitions of the reported metrics and hence the results
should correlate with other measurements of the the same variables.

The Low Level programs are obsolete and need to be replaced.  I have
written seven simple programs, with MPI and PVM versions, and offer them
as a replacement for the Low Level suite.

I strongly suggest that we delete or withdraw from distribution the
current Low Level suite.

---

From: Mark Baker <[email-masked]>
Date: Thu, 8 Jan 98 12:10:55 GMT

I am in agreement with Charles and Rolf about the low-level codes.

We've known for some time that they (the codes) are less than perfect,
if not in some cases flawed. At the SC'97 Parkbench meeting it was
mooted that Parkbench should concentrate on producing, supporting,
analysing and recording Low-Level codes and results. If this is the
case then we should certainly ensure that what we support codes
that are soundly written and produce consistent and reliable results. 

I certainly believe that a set of codes, akin to the low-level ones,
should be part of the Parkbench suite. Maybe this is a good time to
replace the current codes with those that Charles has produced !?

As a side issue, I think we should produce C versions of whatever low-level
codes we produce.

Charles, I'd be interested in your thoughts on the codes that Pallas produce 
- ftp://ftp.pallas.de/pub/PALLAS/PMB/PMB10.tar.gz. These are C benchmark 
codes that run: 

PingPong - like comms1
PingPing - like comms2
Xover
Cshift
Exchange 
Allreduce
Bcast
Barrier - like synch1

Obviously, I would'nt like to comment on how well written they are or how reliable
the results that they produce are. I'm relatively impressed with them. I also like
the fact they try and produce results for commonly used MPI functions - 
cshift/exchange/etc. I've run the codes on NT boxes and they appear to produce 
results close to what I would expect. 

Regards

Mark

---------------End of Original Message-----------------

-------------------------------------
CSM, University of Portsmouth, Hants, UK
Tel: +44 1705 844285	Fax: +44 1705 844006
E-mail: [email-masked]
Date: 01/08/98 - Time: 12:10:55
URL http://www.sis.port.ac.uk/~mab/
-------------------------------------