Subject: Revised Minutes

From: "Michael W. Berry" <[email-masked]>
Date: Tue, 01 Jun 1993 17:58:18 -0400

This should be the last revision - please let me know if I have missed
any other changes.  -Mike
-----------------------------START MINUTES----------------------------

       Minutes of the 3rd PARKBENCH (Formerly PBWG) Workshop
       -----------------------------------------------------
       (PBWG= Parallel Benchmark Working Group)

Place: Science Alliance Conference Room
       South College
       University of Tennessee
       Knoxville, TN

Host:  Jack Dongarra 
       ORNL/Univ. of Tennessee

Date:  May 24, 1993

Attendees/Affiliations:
----------------------

       David Bailey,      NASA
       Michael Berry,     Univ. of Tennessee
       Jack Dongarra,     Univ. of Tennessee / ORNL
       Charles Grassl     Cray Research
       Tom Haupt,         Syracuse Univ.
       Tony Hey,          Southampton Univ.
       Roger Hockney,     Southampton Univ.
       Ed Kushner,        Intel SSD
       Brian LaRose,      Univ. of Tennessee
       David Mackay,      Intel SSD
       Joanne Martin,     IBM
       Ramesh Natarajan,  IBM, Yorktown Heights
       Robert Pennington, Pittsburgh Supercomputing Center
       Bodo Parady,       Sun Microsystems
       David Walker,      ORNL
       Patrick Worley,    ORNL

Agenda: May 24, 1993
--------------------

    At 8:36 am EDT, Roger Hockney gave opening remarks and welcomed
all participants to the workshop.  Each participant introduced him or
herself by affiliation and interests.  The minutes of the previous meeting
(Knoxville, March 1-2) were reviewed with two major corrections made to the
minutes: (1) Roger Hockney's name was missing from the Methodology subgroup
list, and (2) a Compiler subgroup (with T. Haupt as leader) should have
been added.  Since the number of attendees was not that large (17),
Roger H. proposed that there be no separate subgroup meetings during
the day and all attendees agreed.

Roger H. then asked the attendees to think about an alternative name for the
group/benchmark suite which was first discussed at the March 1-2 meeting.
The names considered include: PBWG, PARKBENCH, PARABEN, INTERBEN, SIMPLE,
BIGBEN, and INTERPAR.  Roger suggested that formal voting on the new name
be conducted at the end of the meeting (before adjourning).

The initial draft of the group's report was handed out and each
chapter was then discussed in sequence.  Roger H. began the
discussion with Chapter 1: Methodology.  David B. remarked that the
the notation  "Mflop/s" rather than "Mflops" (which was proposed
by Roger H.) is a good standard to adopt.  Although the meaning
of T(p), i.e., elapsed wall-clock time on p processors, was clear, several
attendees pointed out problems with the interpretation of T(1), the elapsed
wall-clock time on 1 processor.  Roger H. pointed out that there can be many
T(1)'s, which leads to confusion in speedup comparisons.  D. Bailey
suggested that T(1) should not have any parallel overhead.  Charles G.
suggested that an efficiency measure based on Amdahl's law be used
as a replacement for speedup.  Ramesh N. suggested that a 2-processor
baseline time be used.  Tom H. pointed out that speedup is important
for compiler performance measurement.  Roger H. agreed that speedup
may be important to report in this particular case.  Ramesh N. felt
that scaled speedup should be computed separately and that the
widespread connotation of the term "superlinear" speedup is
mathematically incorrect.  Charles G. proposed that speedup should 
specifically address caching effects and memory hierarchies.  This 
discussion of speedup ended with David B. agreeing to rewrite Section
1.4.5 (Speedup, Efficiency, and Performance per Node) of the draft and 
address all the concerns mentioned above.

Roger H. reminded the attendees that subgroup leaders are responsible
for their respective chapters of the report (which is targeted for 
release at Supercomputing '93).

Section 1.5 (Performance Database) was the final section of Chapter 1
which was discussed.  Jack D. was opposed to the idea of providing
any graphical display of the on-line benchmarks provided by the PDS
(Performance Database Server) extension to Xnetlib.  Roger H. indicated
that such graphics would make the data more attractive to users.
Michael B. indicated that future PDS development would incorporate a
spreadsheet-based display of the benchmark data from which graphical
utilities could evolve.  Tony H. indicated an interest in designing
a few prototype graphical tools for displaying benchmark data obtained
from PDS.  Michael B. also pointed out that PDS will also provide
SPEC Benchmarks in the future based on discussions with SPEC officials
at a recent meeting in Huntsville, AL.  Jack D. and Michael B. agreed
to rewrite Section 15. and to indicate how to acquire/use PDS.

Before moving on to Chapter 2 (Low-Level Benchmarks), David B. suggested
that the draft include a motivation section which stresses benchmarking
as a science rather than art.  Parallels with other sciences could be
drawn.  David B. was willing to write up this for the report.

Roger H. then led the discussion of Chapter 2 (Low-Level Benchmarks).
He explained the difference between the two proposed timers TICK1 (clock
resolution) and TICK2 (external wall-clock time).  Roger also indicated
that the UNIX timer "etime" is misleading in that does not report elapsed
wall-clock time (reports CPU time instead) and that timer benchmarks
are really necessary in order to understand the meaning of the reported
times.  David B. indicated that he has observed cases in which CPU time
was greater than wall-clock time.  Charles G. proposed that documentation
should indicate that CPU time cannot be reported but also indicate
potential hazards in wall-clock timing (hardware and network errors).
David B. agreed to write a paragraph for the report which would address
these concerns.  Most attendees agreed that several runs of each benchmark
should be made and Roger H. proposed that the minimum time be reported
(rather than an average) for the low-level benchmarks.  The consensus
was unanimous on reporting the minimum time required but Bodo P. pointed
out that operating systems will need to be somehow quantified for
these times.  Tony H. questioned whether or not optimizations should also
be allowed for these particular benchmarks.

A discussion of the Linpack benchmark (Section 2.1.3) and Livermore Loops
(Section 2.1.4) was then initiated.  Roger H. suggested that the
Linpack (n=1000) benchmark be considered as a kernel benchmark.  Charles
G. did not support the use of the Livermore Loops for measuring cache-based
microprocessors, while Roger H. supported their use for measuring the
range of performance on a node (instability).  Tony H. questioned why
the group include sequential benchmarks for a parallel benchmark suite.
He suggested that the report could reference the serial benchmarks
(Linpack, Livermore Loops, SPEC) but should not include them in the suite.

Roger H. then reviewed Section 2.1.5 which discusses the "N sub one-half"
and "R sub infinity" performance measures.  A routine RINF1 from
the Genesis benchmarks could be used to determine these measures.
Roger H. also proposed that memory-bottleneck benchmarks (POLY1, POLY2)
be included (see Section 2.1.6) in the suite.  Whereas vectors
would fit in cache with POLY1, they would not fit in cache in POLY2.
With regard to Arithmetic benchmarks (Section 2.1.7),  Jack D. 
stressed that 64-bit arithmetic be used but Ed K. pointed out that
32-bit is commonly used in many applications (e.g., seismic codes).
David B. proposed that the methodology should encourage 64-bit
arithmetic but not exclude 32-bit in cases where it is explicitly
required (and documented).  Discussions were then curtailed for a short
coffee break (10:20-10:45am).

After the coffee break, discussions concerning Chapter 2 continued.
Patrick W. questioned the type of communication (arbitrary or nearest-
neighbor) that should be used for the COMM1, COMM2 benchmarks for
measuring communication (Section 2.2).  Charles G. enquired as to if
one could measure hidden latency?  Patrick W. suggested that a
protocol be defined and Roger H. responded with the proposal that
"nonblocking send" and "blocking receive" be used.  He suggested
that other variations could be used in optimizing basic routines.
Roger H. asked if matrix transposition really measures bisection bandwidth?
David W. indicated that it does provided the matrix has only 1 data
distribution.  Tony H. suggested that the group think about alternative
benchmarks for measuring bisection bandwidth.  David W. suggested that
MPI communication routines (broadcast, gather, scatter, etc.,) be
used.  David W. will provide information on these routines.  Tony H.
questioned the need for the separate communication bottleneck 
benchmark (POLY3, Section 2.2.4), but Roger H. maintained that it is
best to have it separated from POLY1 and the COMMS benchmarks.
Roger H. pointed out that the synchronization benchmark (SYNCH1)
was missing from Table 2.3.  Patrick W. pointed out that this particular
benchmark will be extremely machine-dependent.  Roger H. suggested 
that the basic "barrier" paradigm be used.  This concluded the
discussion of Chapter 2 on Low-Level Benchmarks.  

Roger H. then asked Tony H. to lead the discussion on Chapter 3 (Kernel
Applications).  Tony passed out his draft of the chapter (not included with
the chapters originally handed out by Roger H.) and reviewed its contents
with the attendees.  For the matrix benchmarks (Section 3.2.1),
Tony H. proposed that the kernel A=B*C be provided and that the group
consider appropriate validation tests based on generated matrices or
input datasets.  It was also stressed that the matrices B and C start
distributed and stay that way.  Tony H. discussed the availability
of a matrix diagonalization code (Intel i860) that could scale
the computation per node.  He will make the code available (from Daresbury
Lab) for review purposes.  Jack D. proposed that routines from SCALAPACK
be used for the dense LU (with pivoting) benchmark and that
an iterative solver for nonsymmetric linear systems be included.  Michael
B., Jack D., and Patrick W. agreed to work on an appropriate sparse 
linear system solver or eigensolver for the suite.  Jack D. suggested
that a Cholesky factorization routine was not necessary as long as
QR factorization was included.  He also stressed that the suite use
state-of-the-art algorithms for each benchmark.  All attendees agreed.

The discussion focused on what type of Fast Fourier Transform (FFT)
benchmarks (Section 3.2.2) the suite should contain.  Bodo P. suggested
that they be structured like the Linpack benchmarks and questioned
whether or not they should be ordered?  David B. suggested that the
1-D FFT should be very large (order of 1.E+06) and need not be ordered.
As an alternative, David suggested that the benchmark really be a
convolution problem to be solved any way desired.  Patrick W. then
questioned whether or not a power of 2 should be used, and David B.
responded that it should be a power or 2.  Ed K. suggested that a
2-D FFT is not needed if a 3-D FFT is provided.  David W. suggested that
there be forward/backward FFT's which are easy to validate.  Charles G.
and David B. agreed to work on the FFT benchmarks. 

For PDE benchmarks (Section 3.2.3), there was a general agreement to
drop Jacobi and Gauss-Seidel from the list of candidate algorithms.
Tony H. suggested that an SOR-based routine from the Genesis benchmarks
be used.  David M. indicated that he could provide a Finite Element
Method (FEM) code but that it might be better to consider it as a
compact application rather than a kernel.  Bodo P. and Tony H. proposed
that such a benchmark be for a 3-D problem.  There was somewhat of
a consensus that there be a single problem and multiple algorithms
provided.

The discussion on Chapter 3 concluded with consideration of other
possible kernel benchmarks (Section 3.2.4).  Patrick W. questioned
whether or not the Embarrassing Parallel (EP) benchmark (from NASA)
should be a compact application.  Roger H. suggested that there
be an integer sort kernel and perhaps a Particle-In-Cell (PIC)
kernel that might be commonly used in domain decomposition applications.
Other suggested kernels (proposed by various attendees) included:
operation counts, intrinsic operations, out-of-core solvers,
check-pointing.  Tony H. indicated that he could obtain an I/O
benchmark from Daresbury Lab.  David B. pointed out that timing
events such as loading is more appropriate for compact applications
than for kernel or low-level benchmarks.

Roger H. then asked David W. to lead the discussion of the final
chapter of the current report (Chapter 4, Compact Applications).
David B. asked if various data layouts should be allowed?  David
W. proposed that there should both HPF and message-passing versions
of the benchmarks and questioned if time should be measured from the
start to finish?  Tony H. proposed that the QCD code from the Perfect
Benchmarks be replaced with GAUGE (available in HPF and message-passing).
Tony H. will acquire GAUGE for the Netlib database (currently
listed as the "pbwg" library).  David B. suggested that an N-body 
code be provided and Bodo P. questioned if the gravity benchmark
(Section 4.2.3) is really a kernel rather than a compact application.
Patrick W. indicated that he could acquire a shallow water code that
is public-domain (parallelized NCAR code).  David W. suggested that
other molecular dynamics codes be sought since those in the Perfect
Benchmarks have problem sizes that are too small.  For a potential
geophysics benchmark, Michael B. agreed to check on the use of the
ARCO benchmark for the parallel suite.  For other potential compact
applications (Section 4.2.7), Bodo P. objected to the use of DYNA3D
(originally suggested by Joanne M.), and David M. agreed to check on the
availability of a FEM code.  Michael B. commented that good candidates
would be those having multiple instances (HPF, message-passing, etc.,).
David W. suggested that the group hold off on investigating commercial
codes till the project matures.  David B. suggested that a reservoir
code be included.  Other suggestions included: CHARMM, AMBER, GAMES,
GAUSSIAN90 (all molecular dynamics codes).  Patrick W. suggested 
that a signal processing application be added and Tony H. proposed
that the applications focus under the "Grand Challenge" research
areas.  Charles G. asked how the performance of the compact
applications would be verified.  Tony H. indicated that the RAPS
project typically generates lots of numbers.  How many problems
to run was another question raised.  Bodo P. indicated that the
SPEC folks use the geometric mean of several runs.  Roger H. pointed
out that there should be sufficient numbers reported to be able
to show the performance variation with the numbers of processors.
It is particularly important to detect Amdahl saturation, and any peak
in performance which is followed by a steady decrease. Between 5
and 10 points for each problem size, roughly equally spaced
logarithmically, would usually be necessary to do this
(e.g. 1, 2, 4, 8, 16, 32, 64 and 128, on a 128-node system).
Such detailed measurements have been made without difficulty on
the LPM1 benchmark, and give a clear picture of performance variation
with both problem size and number of processors.

The discussion of Chapter 4 concluded and Roger H. asked Tom H.
to briefly report on the compiler subgroup activities.

Tom H. suggested that the compiler benchmarks should address how
compilers handle data distribution.  Related issues include
the use dynamic memory, communication, and runtime libraries.
He indicated that low-level compiler benchmarks (synthetic) be added
and that there should be a comparison with hand-coded optimizations.

Having completed formal discussions of the current report, Roger H.
then called for a vote on the new name of the group.  By a 10-7
margin, the attendees voted to change PBWG to PARKBENCH (PARallel Kernels
and BENCHmarks).  Jack D. asked if the group preferred a full-day
or two half days for the next scheduled PARKBENCH meeting in Knoxville.
The majority of attendees preferred the single day format, and so
the next PARKBENCH meeting is scheduled to be in Knoxville on
August 23.  Roger H. asked that the minutes be posted (to comp.parallel
Internet newsgroup and [email-masked]). Joanne M. indicated that
she would have information on the Birds-of-a-Feather (BOF) session
for PARKBENCH at Supercomputing '93 at the August meeting.  Michael
B. briefly reviewed the status of the SPEC/Perfect merger and passed
out minutes of that meeting (Huntsville, May 1-13).  Roger H. then
adjourned the official (third) meeting of the PARKBENCH group
at 2:50pm EDT.  A demo of the PDS tool supported by UT/ORNL was
given by Brian L. to a few of the attendees till approximately 3:15pm EDT.

End of Minutes for May 24, 1993
(M. Berry)