Subject: Minutes of last meeting

From: "Michael W. Berry" <[email-masked]>
Date: Tue, 12 Apr 1994 22:25:44 -0400

---------------------------------------------------------------------
Minutes of the Parkbench Meeting  - Knoxville Hilton - March 28, 1994
---------------------------------------------------------------------

List of Attendees:

Michael Berry   (Univ. of Tennessee, [email-masked])
David  Bailey   (NASA, [email-masked])
Jack Dongarra   (Univ. of Tennessee / ORNL, [email-masked])
Myron Ginsberg  (GM/EDS, [email-masked])
Charles Grassl  (CRI, [email-masked])
Ed Kushner      (Intel SSD, [email-masked])
Tony Hey        (Univ. of Southampton, [email-masked])
David Mackay    (Intel SSD, [email-masked])
Bodo Parady     (Sun Microsystems, [email-masked])
Fiona Sim       (IBM Kingston, [email-masked])
David Walker    (ORNL, [email-masked])

At 9:20am, Jack D. opened the meeting and indicated that he would be
the substitute chairman (in Roger Hockney's absence).  Jack asked Mike
B. to read the minutes from the Portland (SC'93) PARKBENCH meeting.
There were no modifications to the minutes.

It was noted that there are benchmark programs yet to be placed in
Tony H. suggested that there be a formal announcement of the benchmark release.
David W. indicated that the CA codes are still evolving.

Jack D. read a statement from Roger about his motion to step down as
chairman.  The status of the SP publication was then addressed.
There is hesitation from Wiley to give up Copyright but they will allow
Internet access in PDS/Netlib.  A similar situation with MPI 
resulted in MPI going with MIT Press and not Wiley (joint copyright with
MIT Press and UT).  For PARKBENCH, Wiley holds the copyright but allows
electronic distribution.  David M. asked about future revision problems?  Jack
said that there should be no problem and that the on-line version will be
the most recent.  The text of statement read by Jack D. on Roger H.'s
behalf is provided below.

                 PUBLICATION OF PARKBENCH REPORT
                 -------------------------------
The first Parkbench Report has been accepted for publication by Wiley
in "Scientific Programming" by Ron Perrott (Editor). Although there can
be no guarantees, Ron states that the aim is for it to appear in 1994.
The version that has been modified for publication is available by
anonymous ftp from 'ecs.soton.ac.uk' (152.78.64.201) under directory 
'pub/benchmarks', file 'parkbench.ps.Z'. Login as 'anonymous' and give 
your e-mail address as the password. The file is compressed postscript,
and needs a large buffer space when printing because of screen dumps.

Jack then asked the subgroup leaders to give status reports.  David B.
indicates that nothing happening has happened lately with the
Methodology subgroup and that the report is fully updated.

Tony H. then reported on the efforts by the Kernels subgroup.
CG has now been included since there was no sparse matrix code in the
PARKBENCH suite.  There was a proposal to replace the Integer Sort (IS)
with the CG kernel but David B. felt that the IS might be kept.  The
proposal for replacement of IS by CG was made by David Mackay and Ed Kushner
(Intel SSD).  Bodo P. felt the use of IS depends on the size of the array 
to be sorted.  A general discussion on the pro's and con's of keeping the IS
benchmark followed.  David B. felt that a particle-in-cell (PIC) code should be 
included and that the IS kernel is not indicative of what is done in a PIC
code.  David B. suggested that a code from Los Alamos be added.

ACTION ITEM: David Bailey will contact Los Alamos about getting a PIC code.

The insertion of a PIC code was judged to be independent of the IS kernel
decision.  Jack D. and Bodo P. were in favor of leaving the IS kernel in.
Attendees agreed to leave the IS discussion for the next meeting.

Tony H. noted that only three kernels were actually available in Netlib:
Dense LU, QR Decomposition, and Matrix Tridiagonalization.  He then presented
some performance results.  David B. said that the latest NAS Parallel
Benchmarks Report can be put in Netlib.  A hardcopy of this report (RNR
Technical Report RNR-94-006) was distributed to all the attendees.  Tony H.
then demonstrated how the PARKBENCH data (stored in PDS) might be
filtered into graphs using various templates.  He showed sample performance
(Mflop/s) graphs of different kernels (e.g., LU, 3D-FFT).  Charles G.
stressed that we have to specify why log-log graphs are used.  Tony H.
indicated that the graphics package would allow the user to choose their
own display options.  Jack asked if we want to include price-performance
graphs?  Bodo P. suggested that the user could supply their own price 
information and Jack suggested that the NAS table of prices be available.
Charles G. suggested that list price might be more reasonable than the actual
price.

Tony H. said that his group wants to be able to express the performance of
kernels in terms of low-level parameters (kernels) and that they are
trying this with the following kernels: FFT, Matrix-Multiply, and SOR.

The question of which plotting package to use was raised.  Possible options
are: Xplot, Gnuplot, and jgraph (UT-CS).  David W. and Jack D. both suggested 
that the user should be able to select the plotting mechanism from PDS 
and have plot displayed on their local workstation but Tony H. wants the
user to be able to modify plots on their local machine.  Attendees felt
both features were worthwhile

The current list of kernels and their versions were then discussed.

1. Matrix Benchmarks (Dense matrix Mult., transpose, dense LU,QR decomposition,
   Matrix tridiagonalization)

   Versions: F77, PVM, HPF
 
Bodo P. asked if we want to use a Strassen multiply code?  David B. asked
if we want to encourage them to use Strassen's method?  Charles G. felt it
should be left out due to memory and accuracy concerns.

2. Fourier transforms (1DFFT,3DFFT)

    Versions: F77,CMF for 3DFFT, NX2

3. PDE Kernels (SOR, MG, SoHo, NAS)

    Versions: HPF,PVM

4. Other (NAS:EP, CG, IS; I/O)
    
Bodo P. stressed that disclosure be encouraged and attendees agreed that
vendors are allowed to use supported libraries (available to customers).
David B.  asked that a paragraph disclosure of optimization techniques. 
Charles G. pointed out that vendors are liable when they put source codes out
that were not really supported by the vendor.  

Attendees agreed that baseline runs be done first followed by
optimizations in which the integrity of the "driver" is kept, i.e.,
guarantee that the same problem and accuracy is obtained
(re: pages 17-18 of PARKBENCH report).

ACTION ITEM: Jack D. coordinate the specification of optimization procedures
for the kernels and their packaging for Netlib.

Jack D. pointed out that their are no official MPI results available.  
Tony H. said the kernels should all be available for release at the end
of the month (April).  PVM versions of the low-level benchmarks will also
be included.  The discussion of the kernels concluded and Jack D. then
asked David W. to report on the efforts of the Compact Applications (CA)
subgroup.

David W. pointed out that he has a Fortran-M version of a PIC code that he
might be able to get converted into a Fortran+MP code.  Tony H. indicated
that such a code might be better suited as a CA rather than a kernel.

ACTION ITEM: David W. will see if his PCI code can be converted into
a Fortran message-passing program for PARKBENCH.

Tne current list of submitted CA's is given below:

POLMP (Fluid Dynamics)
SOLVER (QCD,CG)
GAUGE(QCD,MC)  (not available right now but easily obtained)
SHALLOW H2O (ORNL)         Versions: PICL
ARCO (Seismic Migration)   Versions: YAMPOR, PVM
LU,SP,BT (CFD from NAS)    Versions: NX2, F77, CMF, Cray Multitasking

Note: HPF versions of first 3 do exist.

CA's to be added are listed below:

MOLECULAR DYNAMICS (ORNL)   Versions: F77
PIC (LPM1 or LANL)          Versions: PARMACS, PVM

It appears that about 10 CA's can be collected for PARKBENCH.
Group discussed if this is a satisfactory list and pondered any redundancies.
Tony H. asked that application experts review the list of codes.
Myron Ginsberg stressed that some FEM codes be included but Bodo P. and
Tony H. pointed out problems with obtaining public-domain codes for 
applications like "crash testing".  Myron G. noted that large structures or
aerodynamics are the key applications used in the auto industry.

ACTION ITEM: Tony H. he will try to find a public-domain crash code. 

Charles G. asked what application areas are missing?   Bodo P. has a
prime factorization code that could be added.  Fiona S. (IBM) pointed 
out that the list of CA's does not include a solid state Chemistry code.
Tony H. mentioned that there is a GAMES code that might be considered.

ACTION ITEM: Tony H. will ask Martin Guest about availability of GAMES-UK
or AMBER benchmarks.

David W. indicated that codes satisfying the constraints listed in Chapter
5 of the PARKBENCH report should be accepted.   Attendees felt that codes
solving the same problem be left in the suite.  Myron G. asked if an on-line 
transaction processing could be added? Tony H. indicated there is sensitivity
by those groups (financial community) in reporting results.  Mike B.
indicated that the group avoid averaging the results of the CA's (such
as the geometric mean).

Jack D. then suggested that the group break for lunch and restart at
1:00pm EST.

After lunch, a discussion of verification procedures followed.
Jack D.  will produce verification procedures for kernels; David B. will
provide such procedures for the NAS Parallel Benchmarks used in PARKBENCH.
Charles G. pointed out past problems in the Perfect effort's verification
problems (changing arithmetic, different algorithms?).   He indicated that
there would great interest to include residual information (error analysis).
One could track some residuals for different numbers of processors.  David B.
suggested the group should reserve the right to verify submitted PARKBENCH
results.  No specifics on how the verification would be conducted would
be written down -- this was a unanimous decision.

Jack then noted that PVM to MPI conversions are in the near future.  He then
asked the group to comment on the relationship with the SPEC/HPC group.
Tony H. provided the details of a conversation he had with David Kuck.
Kuck put SPEC/Perfect "back on track" at January SPEC meeting.
SPEC/HPSC apparently has to pay their own way. They have ARCO, NAS Parallel 
Codes, Weather code from Intel, Canadian weather code.  Charles G. pointed out
that their press release was the result of a "contentious meeting" in
California.   Fiona S. questioned that their was any firm decisions from
that meeting.   Charles G. pointed out that vendors are tired of having to
maintain yet another benchmark infrastructure.  Ed K. indicated that
Intel is interested in shared-memory type parallel processing with
putting several CPU's on a single chip.  It's not clear how Intel SSD relates
to SPEC.  Intel SSD provided $5K to HPSC (at management level).  
Ed K. on the behalf of Intel SSD (w/ backing of IBM and CRI) will invite
Jack to give talk to SPEC/HPSC group.  Fiona S. was not comfortable with
the size/number of codes HPSC is wanting to consider.  There appears to be
conflicting attitudes in SPEC/HPSC in allowing optimized codes or strictly
baseline (no touch) codes.  Ed K. will contact Dave K. to see if Jack will be
invited to talk to HPSC.  Mike B. agreed to give a talk to SPEC/OSSC on Wed
morning about the PDS/Netlib infrastructure  [At that meeting, SPEC/OSSC
voted 9-0 in favor of posting SPEC Newsletter data with a 3-month delay].

All attendees agreed that Roger H. should remain chair of PARKBENCH.
Jack then asked if the SC'94 meeting should be a BOFS or a roundtable
discussion.  Group decided that the meeting would be best suited as a BOFS.
April 4 is the deadline for scheduling a BOF/roundtable discussion for SC'94 in
Washington DC.  

ACTION ITEM: Jack D. will apply for a PARKBENCH BOFS on Roger's behalf.

Plans for the next meeting: first release of the benchmarks, complete set of
CA's and some initial results.  Video conferencing is a possibility for the
next meeting.  Jack D. indicated that either CLI or PICTURE-TEL technology
can be used.  A third alternative is to use the Mbone/Internet, but the
transmission is 2 frames per second. The group felt an experimental video
conference would be interesting.  Tentatively the next meeting is set for August
here in Knoxville - Monday, August 29.  In June, various video technologies
will be tested (MBONE is marginal in expectations). 

NEXT MEETING: Monday, August 29 in Knoxville.

Jack D. then asked Tony H. to talk about the most recent RAPS Workshop 
(RAPS = Real Applications on Parallel Systems).  It is supported by IBM,
Intel, TMC, Fujitsu,  Cray, ACRI, and Convex.  Members get privileged access
to codes: AVL (FIRE code), ESI (PAMCRASH code), ECMUF (IFS code).  The
workshop was held in December, 1993.  A talk on 3-D IFS was given -- 
HPF applied to FIRE code.

Tony H. also spoke about the Europe HPCN Initiative which is funded to
port commercial codes to MPP's (generic parallel versions with PVM and
PARMACS not HPF).  He detailed some of their work in the published
"Europort" reports:

Europort-1: Focus on CFD and FEM [PAMCRASH, NASTRAN].
Portable and scalable codes desired and made publicly available.
Basically part of ESPRIT project.

Europort-2: Focus on Computational Chemistry, Databases, Oil Reservoir,
Electromagnetics, Radiotherapy, Earth Observation, Drug Design,
Visualization.  Oil application code is PROCLIPSE (ECLIPSE100).
Comp. Chemistry codes included GAMES-UK, ADF, VAMP, GROMOS, MNDO, TURBOMOLE.
Other applications/codes are Electromagnetics [TOSCA,ELEKTRA],
Radiotherapy [EGS4], and Databases [ADABAS].

Tony H. listed contacts for these efforts and noted that the next RAPS
Conference is in Hamburg, Germany on May 26-27:

Europort-1: C. Thole, GMD, Sankt Augustis
Europort-2: A. Colbrook, Smith Systems Engineering

Jack D. then asked about the creation of a "comp.parallel.benchmark" 
or "comp.parkbench" newsgroup.  Jack indicated that an electronic
call of votes (over the Internet) would be required to create such a
newsgroup.  Charles G. supports the creation of the newsgroup over mail 
reflectors.  Group felt that "comp.parallel.performance" was the better
choice for the name and that such a newsgroup should be unmoderated.

Jack D. then suggested that there be a PARKBENCH announcement to HPCWIRE 
when all the codes are tested and assembled into Netlib.  Mike B. then
reported on the status of Mosaic version of PDS.  He indicated that the
current URL for PDS/Mosaic is
http://netlib2.cs.utk.edu/performance/html/PDStop.html.

Jack D. adjourned the meeting at approximately 3:15pm EST.