Subject: minutes

From: "Michael W. Berry" <[email-masked]>
Date: Mon, 06 Sep 1993 14:50:01 -0400

Here are the minutes of the last meeting which were posted to
comp.parallel.  -Mike

       Minutes of the 4th PARKBENCH (Formerly PBWG) Workshop
       -----------------------------------------------------

Place: Science Alliance Conference Room
       South College
       University of Tennessee
       Knoxville, TN

Host:  Jack Dongarra
       ORNL/Univ. of Tennessee

Date:  August 23, 1993

Attendees/Affiliations:

                Michael Berry, Univ. of Tennessee
                Philip Tannenbaum, NEC/SPEC
                Ed Kushner, Intel
                David Mackay, Intel
                Charles Grassl, Cray Research
                Bodo Parady, Sun Microsystems
                David Bailey, NASA
                Jack Dongarra, Univ. of Tennessee/ORNL
                Tom Haupt, Univ. of Syracuse
                Tony Hey, Univ. of Southampton
                Joanne Martin, IBM
                David Walker, ORNL

The meeting started at 9:10 EDT with Tony Hey chairing the meeting for the
absent Roger Hockney.  Tony asked if there were any changes to the minutes
of the May Parkbench meeting.  There were no changes suggested by the attendees
so the minutes were accepted.  Tony then lead the group through the current
draft of the Parkbench Report to be distributed at Supercomputing '93 in
Portland.  Each attendee was provided a copy of the current draft along with a
a handout containing modified sections written by David B.

David B. suggested that his 0.1 Philosophy section be inserted before Roger's
text in Chapter 1 (Introduction).  David B. stressed that reproducibility be
mentioned in the current draft.  Phil T. pointed out that Perfect's new focus
allows optimized versions to eventually become new baseline numbers.  Tony H.
felt that Parkbench should have some sort of relationship with SPEC/PB.  Tom H. 
said that he would provide a few sentences for compiler benchmark development
in the Introduction.

Tony H. then motioned that the group move on to discuss Chapter 2 (Methodology).
David B. stressed that CPU time has some value but that the pitfalls in its use
should be included in Section 2.2 (Time Measurement).  David B. will merge his
section 0.2 in with Roger's Section 2.2.  Charles G. pointed out that one cannot
really define minimum performance (worst case).  The group then discussed what
methodology for measuring elapsed wall-clock time should be used.  Charles G.
offered the following 4 options:  (1) make 3 runs and take the minimum, 
(2) make 3 runs and the maximum, (3) make 3 runs and take the arithmetic
average, and (4) say nothing but recommend that 3 runs be made and ask for 
complete details on how the reported time was obtained.  The group agreed that
option (4) would be most appropriate for ParkBench at this time.  Charles G.
also suggested that TICK1/TICK2 essentially measure the accuracy of the timers. 
Ed K. explained a timing problem observed by Paragon users after reboots -- he
indicated that there should be flexibility in reported timings so that
"typical execution time" actually be stored in the Parkbench database. 
David B. agreed to insert a discussion of reproducibility into
Roger's Section 2.8 (Benchmarking Procedure and Code Optimisation).  Charles G.
asserted that there should also be some emphasis on how runs were timed with 
respect to sensitivity.

Tony H. then led the discussion on to Section 2.3 (Units and Symbols)
and questioned whether or not this section should really be in an Appendix?
David B. recommended that the Section be left where it is and most of the other
attendees concurred.  Mike B. asked the group to change "msend" to "send" on
page 4 of the report following Roger H.'s suggestion.  Moving on to Section 2.4 (Floating- Point Operation Count), David B. felt that the "8 flop" listed for
"exponential, sine etc." was not realistic.   He and Joanne M. felt a more
correct assessment might be "20 flop".  Phil T. pointed out that the Cray HPM
was used for flop counts of the Perfect Benchmarks.  Bodo P. asserted that
since caching is typically used to evaluate many
intrinsic functions, reported counts might be distorted.  Tony H. stressed that
counts are needed for Mflop/s ratios that will ultimately be used.  Jack D. 
indicated that he would be willing to contact J. Demmel and W. Kahan for expert
advice on what flop counts should be associated with the various operations --
all attendees agreed that this should be done.  David B. and Joanne M.
suggested that there should be standard flop counts that are in some sense 
"close" to those for a real machine (eg., Cray).
Charles G. suggested that a bandwidth parameter be included  -- theoretical
data-access rate, for example.  David B. agreed to modify Section 2.4 to
include a memory parameter, and then asked whose responsibility is it to produce
flop counts for the Parkbench codes?  He suggested that a formula be given for
the Kernel benchmarks and perhaps Cray HPM counts for the Compact-Application
benchmarks.  Tony H. pointed out that standard flop counts should be provided
for each problem size.

The discussion then turned to Section 2.5 (Performance metrics).  David B. 
indicated that his Section 0.3 would replace Section 2.5.5 (Speedup,
Efficiency, and Performance per Node) on page 9 of the draft.  David B. also
suggested that there be no speedup statistic kept in the Parkbench Database
(PDS).  Phil T. suggested that David B.'s points (4) and (5) on uniprocessor 
times and speedup when the problem is too large to run on a single
processor/node be combined with his (3) for a general
policy on speedup statistics.  Charles G. stressed that efficiency is important
and speedup relative to Amdahl's Law be considered.  David B. suggested that the
discussion of speedup be limited to avoid controversy.  Jack D. and Michael B.
agreed to provide a few postscript images of PDS for a Appendix to the Parkbench
report.  Bodo P. suggested that the adjective "obsolescent" be inserted before
"SPECmarks" on the last line of page 10.  SPECfp92 and SPECint92 are the
current SPEC benchmarks reported.   With regard to Section 2.7 (Interactive
Graphical Interface), Tony H. indicated that the Southampton Group would 
provide the graphical front end for the Parkbench database.
Bodo P. asked that a single output format be developed
for inclusion in documents (e.g., embedded postscript for LaTeX documents).
Jack D. indicated that each subcommittee will be responsible for providing
codes to the Parkbench suite.  Phil T. questioned whether optimization of codes
(Section 2.8) should be limited, e.g., use high level languages only?
Joanne M. responded that there should be no restrictions.  Tony H. pointed out
that vendors will use a variety of tactics to optimize kernels and low-level 
benchmarks, and that compact applications will most likely require different
forms of optimization.  David B.  insisted that any library routines used 
should be found in any vendor-supported library (i.e., could at least be
purchased).  Tony  H. reminded the attendees that there should ultimately be 3
versions of each benchmark: Fortran90, PVM/MPI, and
HPF.  Phil T. questioned whether or not these versions would follow any
"standards".  Jack D. indicated that MPI (or HPF) would not become any
ANSI-supported standard.

Bodo P. suggested that "XOPEN" has a procedure for creating standards without
going through ANSI-related bureaucracy.  XOPEN is nonprofit British corporation.
A majority of the attendees felt that XOPEN might be an appropriate outlet for
establishing standards for MPI or HPF.  Bodo P. asked if PVM uses shared-memory
function calls? Jack D. indicated that a new version of PVM which uses
shared-memory was under development.  Bodo P. indicated that P4 does support
shared-memory.  Phil T. was concerned that moving biased code to new 
machines would not be fair and suggested that the "as-is" run should simply be
a "minimum sanity run" which comprises minimal changes just to get the code to
run.  Joanne M. and Bodo P.
suggested that the first sentence in the Introduction (Chapter 1) indicate that
"scalable distributed-memory (message-passing)" be the target architectures.
The role of shared-memory architectures was not clear to several attendees.
At 11:05 EDT, the group took a 15 minutes coffee break.

At 11:20 EDT, the meeting resumed with Tony H. leading the discussion of
Chapter 3 (Low-Level Benchmarks) on page 15 of the report.  There were no
comments/changes for Section 3.1 (Introduction) so the discussion focused on
Section 3.2 (Single-Processor Benchmarks) early on.  Charles G. indicated that
the descriptions of TICK1 and TICK2 do not really discuss a timing methodology
and related caveats.  He agreed to rewrite Sections 3.2.1 and 3.2.2 on TICK1 and
TICK2, respectively.  Jack D. suggested that a catalogue of machine-specific
timing routines be maintained.  Bodo P. and Phil T. felt that the use of 
IF DEF's in TICK1 and TICK2 would be sufficient to select the appropriate
machine timing intrinsic function for elapsed wall-clock times.  Jack D. agreed 
to write a paragraph (for Sections 3.2.1 and 3.2.2) describing how machine-
dependent timer calls are handled.  Tony H. asked for volunteers to test POLY1
and POLY2 (Section 3.2.4) and indicated that he would provide the source code.  Tony also indicated that Roger H. would develop COMMS3 and POLY3 for testing
on Intel machines (Sections 3.3.2 and 3.3.3).  For Section 3.4 (Appendix), 
Ed K. agreed to provide more recent data for Table 3.2 (page 22).  Jack D.
pointed out that clock speeds were missing in this table and Tony H. indicated
that Roger H. will contact Ed K. about acquiring more recent data for such
tables.  Tony H. stressed that vendors' approval be granted before listing
such data.

For the Kernel Benchmarks (Chapter 4), Tony H. indicated that he had discussed 
the availability of parallel kernels from TMC (page 28).  Ed K. indicated that
Intel had single node and parallel libraries and mentioned the existence of
routines for banded linear systems, matrix multiplication, and parallel FFT.
Joanne M. indicated that IBM only had serial mathematical libraries.   David B.
pointed out the typo "diificuly" on page 25 of the draft.  With regard to
Matrix benchmarks (Section 4.2.1),
Jack D. indicated that the benchmarks were almost complete -- only support
software (makefiles, input/output files, etc.) is needed.  He specifically
mentioned the availability of matrix transposition, dense matrix multiplication
for multicomputers, routines for reduction to tridiagonal (symmetric) and
hessenberg (unsymmetric) form,  BLACS for PVM, LU and QR factorization.
Michael B.  pointed out that #5 on page 26 should be "conjugate gradient method
for solving linear systems" rather than "eigenvalue problem by conjugate
gradient." Jack D. indicated that both Fortran-77 and PVM versions of the
conjugate gradient method exist.  What is needed for all the above mentioned
kernels include: problem sizes, standard input files, makefiles, and README
files.  Tony H. asserted that the group needs a timescale for future HPF
versions of the kernel benchmarks.
Tom H. agreed to help Jack D. with this.  Jack D. indicated that he would also
solicit the help of Jim Demmel (Berkeley).

For the Fourier Transform benchmarks (Section 4.2.2), several attendees stressed
the need for a large 1-D FFT and the 3-D FFT and PDE by 3-D FFT (#2 and #3 on
page 27) be merged into "3-D FFT".  Tony H. reminded the group that a validation
procedure will be needed for the 1-D FFT and David B. agreed to provide a
convolution problem using a large vector of integers (e.g, 1024 ints).  Using
linear convolution,  checksums would have to agree and the maximum deviation
obtained on any machine could be measured.  Charles G. agreed to help David B.
on this.  For the PDEs (Section 4.2.3), Tony H. indicated that #1 (Jacobi), #2 
(Gauss-Seidel),
and #4 (Finite Element) would be dropped from the current list.  David W. was
unable to acquire the FEM code discussed at the May Parkbench meeting.  Tony H.
agreed to provide the SOR benchmark from the Genesis Benchmarks and David B.
will provide a multigrid benchmark from the NAS Parallel Benchmarks.  These
2 benchmarks do not solve the same problem (SOR for Poisson's Equation and
Multigrid for another application) but several versions of them (serial, MPI,
HPF, etc.) are possible.  Jack D. agreed to write a formal letter to NASA on
behalf of the Parkbench group to request the use of the NAS Parallel Benchmarks.
David B. indicated that these codes are now available for world-wide
distribution.  For "Other" kernel benchmarks (Section 4.2.4), Tony H. agreed to add a description of a candidate I/O benchmark (not mentioned in the current
draft).  Joanne M. and
Tony H. stressed the need for the Embarassing Parallel (EP) benchmark, and Ed K.
felt the Large Integer Sort benchmark should be included as well but questioned
if it should be a "paper and pencil" benchmark similar to the NAS Parallel
Benchmarks.  He indicated that David Culler ([email-masked]) and
Leo(?) Dagum ([email-masked]) could help supply this benchmark code. Tony H.
reminded the group that Roger H. can provide a Particle-In-Cell (PIC) code as a
candidate kernel benchmark also.  At 12:15 pm EDT, the group broke for a 1-hour
lunch.

At 1:15 pm EDT,  Tony H. indicated that he would contact David W. later about
the status of the Compact Applications which comprises Chapter 5 (David W.
had not arrived yet).  Tom H. suggested that when preparing an HPF version of
the kernel for dense matrix multiplication, there are 2 choices: (1) rewrite
the kernel in HPF (as we plan to do with everything else), or (2) use a call
to HPF (actually Fortran 90) intrinsic function MATMUL.  Tom H., Jack D., and
Tony H. preferred the option (2).

A discussion of problem sizes prompted Phil T. to ask if 1 or
2 gigabytes of memory constituted a large problem?  Charles G. pointed out that
scalability is important but not always possible (often algorithms will change
as problem size increases).  Joanne M. suggested the group think about future
problem sizes to anticipate relevant problem sizes a few years from now.  Jack
D.  suggested that 10 gigabytes defines a large problem and that 1-2 gigabytes
would be a medium size problem.  Small problem sizes would be needed for simple
testing purposes.  Bodo P. asserted that memory should be specified as part
of the dataset name.  Tony H. summarized that there should 3 problem sizes:
(1) test problem size, (2)  moderate size problem, and (3) grand challenge
problem.   All problem specifications would be denoted in README files.

At this point, a discussion of the Compiler Benchmarks (Chapter 6) was lead by
Tom H, who suggested that the Introduction (Section 6.1) be merged with the
Introduction (Chapter 1).  He suggested that a new metric for the compiler
benchmarks also be mentioned in this introduction: "ratio of execution times of
compiler generated to hand-coded programs as a function of the problem size and
number of processors engaged."  Since there were no comments for Section 6.2
(HPF), Tom H. proceeded on to Section 6.3 (Benchmark Suite) and specified that
no multi-statement optimization is addressed in the current compiler benchmark
set.  Tony H. asked if the name of these particular benchmarks should be changed
to "HPF Compiler Benchmarks."  He also suggested that an Appendix discussing
MPI and HPF be included in the report.  Tom H. then reviewed the Description of
Codes in Section 6.4.  He indicated that all were existing codes and that the
last 3 involve FORALL statements while the las 4 involve array distribution.
Bobo P. noticed that subroutine in-lining was missing from the set.  Ed K.
indicated that Intel would eventually have HPF version of all the codes listed.
He also suggested that there be HPF compiler benchmarks more suitable for the
compact application benchmarks since the current set of compiler benchmarks seem
for appropriate for the kernel benchmarks.  Jack D. asked just how many HPF
compilers would be available by the time for Supercomputing '93 in November?
Tom H.  felt there would be none but Ed K. said there would be at least one.
Jack D.  suggested that the codes (currently in Section 6.4) be removed from the
report.  A description of them would suffice.  

At 1:45 pm EDT, David W. arrived to the meeting and Tony H. asked him to
report on Compact Applications (Chapter 5).  David W. indicated
that there were no codes submitted since the May Parkbench meeting.  He also
suggested that there be an application form for submitting compact application
benchmarks.  Such a form would specify the total number of flops required and
memory requirements, and could be posted to the Usenet News Group 
"comp.parallel."
David B. will submit the NAS Parallel Benchmarks and David W. indicated that
a Shallow-Water code and a Molecular Dynamics code would be available.  Tony H.
indicated that he could provide a "real" QCD code and then asked about the
status of the FEM code discussed at the May meeting.  David M. said that
particular code was not readily available. Joanne M. suggested
that 5 compact applications would be sufficient.  David M. indicated that he
would check again on the use of the FEM code (in the public domain).  Tony H.
asked David W. to contact Chuck Mosher about the ARCO suite (Joanne M.
indicated she had Mosher's electronic mail.  David B. asserted that an N-body
problem might be an appropriate compact application.  The current list of
compact applications considered by the group include:  (1) ARCO suite, (2) NAS
CFD benchmarks, (3) QCD from Southampton, (4) Shallow Water code from ORNL, and (5) Molecular
Dynamics code.

Tony H. then suggested that a Summary of Actions be reviewed so attendees agreed
on the remaining work to be done.  In preparation for Supercomputing '93 in
Portland, the following benchmark suite should be assembled:

	PDS/Xnetlib (UT)
	TICK1, TICK2, RINF1, POLY1, POLY2 (Roger H.)
	COMMS1, COMMS2, COMMS3, POLY3, SYNCH1 (Roger H.; need PVM versions)
	PUMMA, LU, QR, CG, TRDI (UT/ORNL supplies seq. and PVM;
	                         Syracuse provides HPF)	
	FFT 1-D, FFT3-D (David B. - NASA supplies seq. and Intel versions)
	SOR, Multigrid (seq., MPI, HPF provided by Tony H. and David B.)
	EP, Integer Sort (seq. provided by Tom H. - Syracuse)
	I/O (provided by Tony H. - Southampton)
	HPF Compiler Benchmarks (Tom H. - Syracuse)
	QCD (Tony H. - Southampton)
	CFD/NAS(3) (David B. - NASA; seq., MP, CMF)
	Molecular Dynamics (David W. - ORNL; seq., PICL)
	ARCO Suite (2) (Jack D. and David W.; seq. and PVM)
	Shallow Water (David W. - ORNL and Tom H. - Syracuseg 
                
Each code is to have 3 input sets as described earlier.  UT/Netlib will
collect/archive all candidate benchmarks  for the Parkbench suite.
Makefiles, README files, input files, etc. should be provided.

Tony H. then called for a discussion on future actions.  Tony suggested that
subcommittee leaders put revised sections on the net, and Michael B. agreed to
assemble final draft (with the help of Roger H.).  Revised text should be mailed
to [email-masked] by October 15, 1993.  Joanne M. indicated that
Supercomputing '93
runs from Tuesday to Friday the week of November 17, and the current tentative
evening session for Parkbench is Wednesday, November 17, 4:30 pm.  Tony H. asked
for opinions on future chairmanship (on the behalf of Roger H.).  All attendees agreed that Roger H. was best suited to fulfill this role for the immediate
future.  There was strong opposition to the discussion of future chairmanship
at the open meeting in Portland.  For future meetings, Joanne M. and Tony H.
suggested that the group explore "videoconferencing" between a US site and a
European site.  Bodo P.  suggested that such meetings should be coordinated with SPEC.  Jack D. offered to
host a joint Parkbench/SPEC meeting for early 1994.  Bodo P. then reviewed the 
organizational structure of SPEC for the attendees.  He also mentioned a new
SPEC benchmark called "PAR93" which comprises a 512 MB problem size for
under 60 processors with automatic parallelism, shared-memory, and
compiler/system tuning allowed.  After Bodo P. completed the SPEC overview,
Tony H. asked the group whether Parkbench should ultimately be sponsored by 
SPEC.  At this time, Phil T. gave a short lecture on the evolution of
SPEC/Perfect.   Joanne M.  suggested that Parkbench should be separate from
SPEC/Perfect (may be called
SPEC/HPSC - High Performance Steering Committee in near future) at least during
the initial development phase of the benchmark suite.  Phil T. mentioned that
the current application benchmarks for SPEC/HPSC include ARCO (fdmod, fkmig,
seis), LANL (false, pueblo), IBM (turb3d), and ETH/Zurich (NCMD - QCD).  All of
these are Fortran-77 versions with only ARCO having a message-passing version
available at this time.  David W. also remarked that it was very premature to
discuss a Parkbench-SPEC/HPSC merger at this time.  Tony H. indicated that
PEPS and Euroben should also be included and asked  Jack D. to formally contact
them (they would like to maintain a copy of the PDS data).  

Some of the final discussions at the meeting concerned the default precision
the codes should be run in for validated results.  Charles G. insisted that the
precision be specified in the code documentation.  Jack D. felt that the kernel
benchmarks should normally be run with 64-bit arithmetic unless otherwise
specified.  Some of the compact application codes (e.g., ARCO suite) may be
run in 32-bit precision, however, to reflect "real" use in industrial
applications.  David B. also pointed out that the precision used during a run 
should be denoted in a code's output.
The meeting promptly adjourned at 4:04 pm EDT.

End of Minutes for August 23, 1993
(M. Berry, [email-masked])