Subject: Minutes of SC'94 BOFS

From: "Michael W. Berry" <[email-masked]>
Date: Wed, 23 Nov 1994 19:44:05 -0500

Please send me all corrections to the following minutes of
the joint HPSC/SPEC-PARKBENCH BOFS.  Thanks, Mike

--------------------------------------------------------------------------
Minutes of SPEC/HPSC - PARKBENCH BOFS at Supercomputing'94 on Nov. 15, 1994
(Washington Convention Center, DC)
--------------------------------------------------------------------------

(Attendance list not available)

Roger H. started the meeting at 5:40pm.  Roger handed out a glossy brochure
for the PARKBENCH effort.  Roger mentioned that 10 benchmarks are now
available and the report has been published in "Scientific Programming".
Roger had some extra copies of the reprints for those mentioned in the paper.
A joint workshop of PARKBENCH/PEPS/EUROBEN has been scheduled at the
Univ. of warwick, UK for Dec 15-16, 1994.  Send email to [email-masked]
for more information or to submit papers.  This meeting is organized by
Prof. Nudd at Warwick.

Tony H. expressed warm thanks to Roger H. for his efforts in guiding the
PARKBENCH effort.  Michael B. read the action items from the August
PARKBENCH meeting in KNoxville and it was noted that the suite still
required a very long 1-D FFT - to be provided by D. Bailey.

Tony Hey (new chairman) then overviewed the current Parkbench effort.  Tony
noted the 2 year anniv. of the Parkbench effort.  Tony reviewed the objectives
and the deliverables of the Parkbench effort whose focus is on distributed
memory-based benchmarks.

Tony reviewed the 5 subcommittees - he indicated that perhaps we should have
an "analysis" subcommittee.  Tony then briefly reviewed the actual benchmarks
in the PARKBENCH suite: low-level, kernels (all available including the
NAS Parallel benchmarks), and compact applications (SEIS1.1,POLMP,SOLVER,PSTSWM,
3 CFD codes from NASA).  It was noted that POLMP results have already been
published in 3 papers. Tom Haupt's HPF compiler benchmarks were also
mentioned.

Tony mentioned the PARKBENCH URL for source codes, reports, documentation.
He indicated that one can now view results via PDS and GBIS.  He then
demo'ed the GBIS graphical interface.  This GUI isfor both Genesis and 
PARKBENCH results.  Many formats for outputs are available.  Several
of the URL's for GUI and PDS access were listed:

PARKBENCH desc & report -- http://www.epm.ornl.gov/~walker/parkbench/
PARKBENCH doc & codes   -- http://www.netlib.org/parkbench/
GBIS (Genesis Benchmark Information Service) 
    -- http://hpcc.soton.ac.uk/RanD/gbis/papiani-new-gbis-top.html
PDS -- http://netlib2.cs.utk.edu/performance/html/PDStop.html

Tony H. noted that one can get benchmark information with links to NSE 
and PDS from the GBIS pages.  He then demo'ed selections for the EP PARKBENCH 
code.  He indicated that graphs can be modified or accept the default graphs.
One can produce either linear or log scales on the axes.  

Regarding Future PARKBENCH work, Tony H. listed the following items.

- parallel version of 1-D FFT

- investigation of suitable codes for I/O kernel

- code uniformity (standard makefiles, etc.)

- reorganize the server repository (release methodology etc.)

- define rules for running the benchmarks

- further results are required (GBIS and for PDS inclusion)

- define a methodology for reporting bugs

- future MPI versions of all codes; benchmarks to test 
  gather/scatter/global sum operations

- shared memory versions (put/get)

- further compact applications coverage

- start up an analysis subcommittee (for low-level kernels)

After concluding the PARKBENCH presentation, Tony H. solicited questions
from the audience.  Here are some of the dialogues.

Q: How many machines reported?  Rather sparse right now -- depends on what is
   available.  Roger H. pointed out that database must be expanded in order 
   for an analysis group to have enough to work with.  He stressed the need
   to understand the performance "surface".

Q: Are sequential versions available?  Yes, there are seq. and parallel
   versions.

Q: Has the PDE2 code essentially replaced the NAS multigrid code?  Yes, codes
   that are more widely used are preferred.  Tony H. indicated that a
   formal process for including/deleting codes is needed.

Sia Hassanzadeh from Sun Microsystems then overviewed the SPEC/HPSC effort.
He provided 2 handouts: 1 on SPEC and 1 on HPSC.  Two committees exist - OSSC
and HPSC.  Two categories of affiliation were described and current membership
total is about 58.  There are regular members ($5K fee) and associate
members ($1K).  He pointed out that SPEC/HPSC  is a nonprofit organization 
whose mission is to establish, maintain, and endorse a suite of benchmarks.
He distributed a handout "HPSC: A Progress Report" to al the attendees.
He then reviewed HPSC's scope (broad range of machines and applications
with discipline-specific suites).

The SPEC/HPSC target application areas mentioned were:

Seismic Processing (ARCO Suite 1.1)
Computational Chemistry (GAMESS)
CFD (TURBO3D)
Enviromental and Climate Modeling
Structural Analysis (FEM)
Information Processing (DB, text retrieval, video-on-demand, etc.)

Multiple data sizes planned and codes should be self-checking.  Sizes can be
small, medium, large and HUGE (ARCO suite uses this for example).  The
programming languages considered are  Fortran77 or ANSI C, PVM/MPI, and the
Shared-Address Space (SAS) programming model.

Sia mentioned that there have been 4 SPEC/HPSC meetings and that Sia is 
Chair and Fiona Sim (IBM) is the Vice-Chair.  They are looking into procedure
for result distribution -- perhaps using PARKBENCH tools/servers.  Run rules
are more restrictive for SPEC/HPSC.

Two of the questions and answers that followed Sia's presentation are
listed below:

Q: What is the optimal number of codes?  A timetable is planned and looking at
  emerging technology.

Q: Will applications only be added to suite if it satisfies the list of 
   parallel versions?  SPEC/HPSC requires a serial and at least 1 parallel
   version such as MPI or HPF.  Code authors or code managers will have to 
   take responsibility to produce follow-up versions.

At 6:30pm, Tony asked for an open discussion on future directions for both
efforts (PARKBENCH and SPEC/HPSC).

Tony H. asked Jack D. for comments about the LINPACK results method.  Jack is
very trusting of the submissions and he sometimes verifies them on his own.

David B. was asked to comment about NAS Paralell benchmark methodology. In
that case, one has to include self-checking code with benchmark.  Sia H. 
indicated that there should be a review process (done within SPEC). Have to 
report a global set of metrics for each code and then provide breakdown of
certain activities like throughput, I/O, etc.

Chuck Mosher (ARCO) indicated that politics of vendors gets to be a problem with
reported results.  Rudy Eigenmann (Illinois) asked about how to handle 
optimized versions of codes.  He indicated that vendors tend not to want opt.
results.  Tony H. indicated that PARKBENCH stresses listing of optimization
and that performance is measured against a canonical mflop count.  Getting the 
actual opt. code is not necessary -- libraries used must be available to
consumers.

Sia H. pointed out that "as is" codes are difficult to manage.  Have
to be concerned with recording "benchmarker's abilities".  Forcing compliance
on the specifications of optimization is difficult.  Fiona Sim stressed that 
customers really want their own version so opt. should be limited. 

David MacKay stressed that PARKBENCH universities work with SPEC/HPSC's
versions of the codes.  They should be able to use the Assoc. membership
SPEC/HPSC status to achieve this.  Tony H. asked if NAS CFD codes would be 
acceptable to SPEC/HPSC? David M. indicated that vendors already had their
own versions.  PARKBENCH distributed version would be experimental but
vendors would keep their own private versions.  Jack D. agreed to be the
arbitrator of distribution of HPSC codes to PARKBENCH members.
PARKBENCH can perform exploratory exercises in code optimization for
SPEC and delay results on the WWW.  

Tony H. then asked Tom H. to provide updates on HPF benchmarks in PARKBENCH
html pages.  He then asked the audience to voice their opinions on the
concept of an "analysis " group.  Tony then asked Aad van der Steen
(Academic Computing Centre Ultrecht) if he would be willing to lead such
a group.  Aad agreed and Tony suggested that inquiries be made into joint
European and American funding. Alistair D., Fiona S., and Roger H. were
willing to be on that subcommittee chaired by Aad.

Tony H. adjourned the meeting promptly at 7pm EST.  The next meeting
for PARKBENCH is schedule for the spring of 1995 in Knoxville.