Subject: PARKBENCH Minutes

From: "Michael W. Berry" <[email-masked]>
Date: Tue, 23 Nov 1993 23:10:14 -0500

Please review the enclosed minutes of the PARKBENCH
meeting at Supercomputing'93 and send me all corrections and
ommissions.  

Regards,
Mike
([email-masked])

PARKBENCH Meeting - Supercomputing'93
Nov. 17,1993, Portland Convention Center 

(The list of attendees is appended to the end of this message).
------------------------------------------------------------------------
At 4:33 pm PST. Roger called the meeting to order.
Roger asked for Chapter authors to take 5 minutes to discuss their chapter.
He then listed PARKBENCH's Aims: comprehensive set, focus on parallel
benchmarking, like Linpack, etc.,

David B. then reviewed the methodology chapter of the report - went over
Section 2.1 (Philosophy) through Section 2.6  (Performance Metrics) &
Section 2.7 (Performance Database).   Bodo Parady mentioned problem with
security for Sun with Xnetlib.  Questioned the resolution of the timers -
Xian-he Sun (ICASE) was concerned about variability in floating-point
operations across machines.   Roger responded that all we want is a scaling
factor.  

Roger reviewed the low-level benchmarks chapter.  The Linpack-100 benchmark
was noted as a single node benchmark.  Fred Gustavson (IBM) indicated that
the Linpack benchmark could be improved with better kernels.  Jim Stacey
(Collaboratory Inc.)  indicated that benchmarks like Linpack does not take
into account overheads (some have inlining capabilities and others do not). 
Bodo Parady (Sun) suggested that the SPECbenchmarks
may have an alternative.  Error on equation (3.3) should be "\hat{r}_{\infty]".
Mohammad Zubair (IBM Research, Yorktown) suggested that a synchronization  primitive should be included in COMMS1,2.  Roger H. indicated that one is already
included in the current PARKBENCH suite.

Steve Hotovy (Cornell) -  questioned the specification of the nodes for 
COMMS1,2 (e.g., difference in rings on KSR).
Ramesh Agarwal (IBM Research, Yorktown) thought COMMS3 was a very good idea.
Roger H. reviewed tables of results and indicated they are somewhat out-of-date.  
Tony H. then reviewed the Kernels chapter and noted the range of applications
that the kernels span.  Some are taken from the NAS Kernels and other from
the Genesis Benchmarks.  Matrix benchmarks include the
dense matrix multiply, matrix transpose, dense LU factorization with partial
pivoting, QR decomposition, matrix tridiagonalizations.  Ramesh A. (IBM) asked
about the data structure for the transpose.  Jack D. indicated that a
block cyclic structure is used.  Ian Duff (RAL, England) felt that vendors
should be allowed to
use their optimized library kernels.  David B. asked why have transpose and
matrix multiplication but several attendees thought the transpose was needed.
Regarding the FFT's (1-D and 3-D) David B. pointed out that the 1-D is really a
convolution and that the size would in the millions (long convolution with
with 2 long sequences via FFT).  Fred Gustavson (IBM Research, Yorktown) asked 
about band matrices - why leave them out?   David B. pointed that band matrices are used in the NAS parallel benchmarks.   

Tony H. then reviewed the PDE kernels (Poisson's equation's via SOR and 
multigrid) and other kernels (embarrassingly parallel (NAS), large integer
sort from David Bailey, and an I/O paper and pencil benchmark).
A comment was made about NAS benchmarks not leaving the country and David
B. replied that all clearances have been given.
Horst Simon (NASA Ames) asked if CG from
NAS would be dropped.  Jack D. indicated it is in Compact Applications.

David Walker then reviewed the Compact Applications chapter.  He pointed that
CA's exhibit behaviors that cannot be found with kernels, and
reviewed the specifications for CA's.  Acceptable languages include: F77
and C plus message passing, Fortran-90, and HPF.  One attendee commented
that message passing can be machine-dependent.  David W. indicated
that the mp libraries should be portable.  No assembly code allowed in the
baseline versions.  Fred Gustavson (IBM) asked if library routines
could be included?  David W. suggested that standard intrinsics would be ok.
Mohammad Z. asked about the use of smart compilers?  David B. said that is fine.
Fiona Sim (IBM, Kingston) asked if vendors could use their own mp libraries? 
It was agreed that vendors should do portable version first as baseline.  Some
attendees were concerned if a vendor had to supply all 3 forms of a CA?  David
W.  indicated that at least 1  would be sufficient.   Tony H. indicated that we may want optimized workstation versions to compare with.

CA's submitted thus far include  ARCO, NERC (3D Hydro),
Nat. Envir. Research Council (NERC), Spectral transform shallow water mode
(ORNL), and a QCD (Lattice Gauge) code (Univ. of Edinburgh).

Fred Gustavson (IBM) asked if any simplex (linear programming) should be
added to the CA suite (applications such as transportation scheduling and
optimization).   Bodo Parody (Sun) indicated an NYU code for simplex method
might be added.  Other attendees notes that the use of complex arithmetic
and ODE's was lacking.  David W. finished by providing an email
address and instructions for submitting candidate CA codes.

Tom H. then reviewed the compiler benchmarks.  No questions were raised about
these particular benchmarks.

Jack D. then reviewed history of Netlib and the inclusion of benchmarks and
the results themselves.  He described the Xnetlib system and PDS.  He
provided sample screens of Xnetlib and noted that the PARKBENCH codes and
results would be included in PDS.  Jack indicated which codes are currently
in Netlib and those which would be appearing soon.
Small, medium, and large datasets would also be included.
Rudi E. (CSRD) asked if version control had been considered?  Jack D. said the
index file will have a version number and source files will have timestamps.
Final form not decided on how to deal with valid results.  Rudi E. (CSRD)
suggested that a steering committee should approve the results. 
A few attendees questioned how will vendors approve results (contact person)? 
Ramesh Agarwal (IBM Research, Yorktown) asked if vendor
should provide optimized code?   A description of the optimizations (Diary)
was deemed sufficient.  Horst Simon (NASA Ames) asked about a quality of
ontrol mechanism?   David B. and Jack D. indicated
we should be trusting but erroneous results do come in.  More discussion
is needed.    Bodo P. (Sun) asked if the best or worst results be reported. 
Ed. K indicated that getting baseline codes is Intel's only problem.  Ian Duff
wanted to know how vendors are reacting to PARKBENCH?  Robert Numrich and 
Kevin Lind (CRI) thought there would be no problem.   One other attendee
was concerned about the lack of throughput or multi-user benchmarks.

The review of the first PARKBENCH report was concluded followed by an
acceptance of the minutes from the previous meeting.  Roger then went over
the rest of the agenda items which included
updating the report/codes, generating a new set of results, and
having a 3-month update for PDS. The Southampton group will produce
graphical tools.  Ramesh Agarwal (IBM Research, Yorktown) asked if we should 
limit the number of processors to be a power of 2?  Roger  H. suggested that
at least 5 values for the number of processors be submitted.  Phil T.
(NEC) reminded attendees that 10 gigabytes was acceptable for "large" dataset
size. 

Concerning the format of future meetings, it was agreed that travel
should be reduced.  Ian Duff (RAL) felt a Quarterly Report might be issued.  
Roger H. asked if email was sufficient and most attendees felt is wasn't
since no deadlines could be enforced.  Jack D. announced the upcoming
March 29, 1994 "PARKBENCH/SPEC "meeting in Knoxville. Roger H. asked for
a show of hands on who was going to theHPCN (high perf. computing and networks) meeting?   Most not going.

Upcoming meetings in March 1994 (Knoxville, TN):

March 28 :      PARKBENCH
March 29,30,31  SPEC

Only 3 people in room would be going to Knoxville for the SPEC meeting.
Roger asked that an August 1994 meeting be a teleconferenced meeting.
Fred Gustavson and Joanne Martin will check with IBM about providing 
facilities.

Roger H. asked where should report be published?  He suggested that the
journal "Scientific Programming" be considered since the HPF report
will be published there as well.  This journal by
Wiley was considered it quite favorable by attendees and Roger H. will
pursue Wiley about the report and yearly updates.   David B. suggested that
the current focus be on publishing report first and then looked at data later.

Roger H. adjourned the meeting at approximately 6:30 pm PST.

------------------------------------------------------------------------
List of Attendees (Nov. 17, 1993)

Neil B. MacDonald	Edinburgh Parallel Computing Centre
Ken Hewick		Northeast Parallel Arch. Center
Ramesh Agarwal		IBM Research, Yorktown
Fred Gustavson		IBM Research, Yorktown
Mohammad Zubair		IBM Research, Yorktown
Aad van der Steen	Academic Computing Centre, Utrecht
Kevin R. Lind		Cray Research Inc.
Robert Numrich		Cray Research Inc.
Tony Hey		Univ. of Southampton
Daniel Frye		IBM Kingston
James Stacey		Collaboratory Inc.
Wayne Pfeiffer		SDSC
Luis Almeida		Paravector
Ken Miura		Fujitsu America
Bodo Parady		Sun Microsystems
Mark R. Smith		Cornell Theory Center/IBM
Wei-Hwan Chiang		IBM Kingston
Jack Cole		USARL
David Mackay		Intel SSD
Robert Eller		Arithmetika
Clive Baillie		Univ. of Colorado
Tom Eidson		NASA LARC
Mike Humphrey		SGI
Linton Ward		IBM
Fiona Sim		IBM Kingston
Chuck Mosher		ARCO
Siamak Hassanzadeh	Fujitsu America
Ed Kushner		Intel SSD
Ian Duff		RAL, England
Stefano Foresti		Univ. of Utah
Steven Hotovy		Cornell Theory Center
Wolfgang Gentzsch	GENIAS
Horst Simon		NASA Ames
Nahil Sobh		Saudi Aramco
Roger Hockney		Univ. of Southampton
Michael Berry		Univ. of Tennessee
John Larson		Univ. of Illinois
Rudi Eigenmann		Univ. of Illinois
David Walker		ORNL
Jack Dongarra		Univ. of Tennessee/ORNL
Xian-He Sun		ICASE
Tom Haupt		Syracuse University