Subject: PBWG Minutes (March 1-2, 1993)

From: Mike Berry <[email-masked]>
Date: Tue, 9 Mar 93 17:19:08 -0500

Minutes of the 2nd Meeting of the Parallel Benchmark Working Group (PBWG)
  -------------------------------------------------------------------------

Place: Room 221, University Center
       University of Tennessee
       Knoxville, TN

Host:  Jack Dongarra 
       ORNL/Univ. of Tennessee

Dates: March 1-2, 1993

Attendees/Affiliations:
----------------------

       David Bailey,      NASA
       Michael Berry,     Univ. of Tennessee
       Ed Brocklehurst,   National Physical Lab
       Jack Dongarra,     Univ. of Tennessee / ORNL
       Koushik Ghosh,     Cray Research
       Tom Haupt,         Syracuse Univ.
       Tony Hey,          Southampton Univ.
       Roger Hockney,     Southampton Univ.
       Brian LaRose,      Univ. of Tennessee
       David Mackay,      Intel SSD
       Joanne Martin,     IBM
       Robert Pennington, Pittsburgh Supercomputing Center
       David Walker,      ORNL
       Pearl Wang,        George Mason Univ. / US Geological Survey

Agenda: March 1, 1993
---------------------

    At 1:00 pm EDT, Roger Hockney gave opening remarks and welcomed
all participants to the workshop.  Each participant introduced him or
herself by affiliation and interests.  The minutes of the previous meeting
at Supercomputing'92 (Minneapolis) were reviewed.  Roger suggested that
the group consider an alternative name for the group such as "PARABEN"
or "INTERBEN".  Participants were asked to think about this over dinner
and cocktails that evening.
    Jack Dongarra mentioned that the mail reflector "[email-masked]"
is now in place and can be used as an electronic forum for the group.
Currently there are about 50 persons on the mailing list.  Jack also
mentioned that anyone can obtain all correspondence information concerning
PBWG requests and solicitations by sending the mail message 
"send archive from pbwg" to [email-masked].  To inquire about the
contents of the archive simply send the message "send index from pbwg" to
[email-masked].  It was noted that this meeting was posted in the
the "comp.parallel" users group (USENET).
    Roger Hockney reviewed the 4 subgroups previously defined for the
effort: Methodology, Classification, Compiler-based, Kernels.  Roger
proposed that the participants not break into subgroups due to the
somewhat lower than expected turnout for the workshop.  All participants
agreed and a general discussion on "Methodology" followed.
    David Bailey was then asked to review the "Proposed Guidelines for
Reporting Performance" which he distributed to the group.  [These guidelines
will be made available from the pbwg archive in netlib.]
    Roger Hockney then distributed an article entitled "A framework for
benchmark performance analysis" from Supercomputer 48, IX-2  (March 1992).
He recommended that the group start using correct performance symbols
such as "Mflop/s" for millions of floating-point operations per second
rather than simply "Mflops" which is somewhat ambiguous.  As discussed
in the paper, temporal performance measures such as T(N,P) and
R_T=T^-1(N,P) were also suggested.  Roger pointed out that the
numerator term, F(N), in the performance measure R_B = F(N)/T(N,P)
should be constant and clearly stated.  Here, T(N,P) is the execution
time, R_T is the inverse of execution time, N is the problem size,  P is
the number of processors, and F(N) is the total flop count.
He suggested that the group steer away from measures such as speedup
and efficiency.  David Bailey pointed out that one must
be prepared to change F(N) to match optimal implementations on
different machines.  Jack D. pointed out that one must be careful though
not to allow performance rates which exceed peak performance.  Roger
H. suggested that the performance database might provide plots
of the metrics as functions of N and P.
    Along with the distribution of the paper by Roger Hockney, Tony Hey 
distributed copies of 2 papers on the Genesis Benchmarks: "The
Genesis Distributed-Memory Benchmarks" (Tech Report, Dept. of
Electronics and Computer Science, Univ. of Southhampton, UK), and
"The Genesis distributed-memory benchmarks. Part 1: methodology and
general relativity benchmark with results for the SUPRENUM computer"
(from Concurrency: Practice and Experience Vol. 5(1), pp. 1-22,
February 1993.)  Tony indicated that the Genesis benchmarks will be
sent to Jack D. for inclusion in the benchmark repository in NETLIB.
    Ed Brocklehurst raised the concern that the European PEPS project
methodology might differ from that of PBWG.  He stressed the need for
collaboration among the two efforts.  He then gave a short lecture on
the PEPS project which is a 3-year funded program ($11 million) which
began in November of 1992.  Participants of the PEPS project include:
TS-ASM (electronics firm - France), NPL, Inecs Fistemi (Italy),
Simulog (France), Sosip (Germany, France, Italy, UK), and Warwick.
The benchmarks supported by PEPS are to be available to the public.
Other concerns of the project involve performance monitoring and
characterization.
     The discussion on Methodology concluded with a comment by Michael Berry
that all accepted codes for the benchmark suite be self-validating.  Roger
H. then asked David Walker to report on the work thus far by the
Classification subgroup.  David W. stressed that the classification of
benchmarks be based on (i.) communication characteristics, (ii.) I/O,
and (iii.) load balance and data layout.   Questions on the inclusion
of profiles and processor utilization data with the benchmarks were raised.
David W. pointed out that benchmarks may range from inherent parallel
codes (e.g., Monte-Carlo) to mesh-based codes or gather/scatter-based 
applications.  David Bailey stressed the need for classifying I/O and
to see what the current measurement of I/O is in existing benchmark suites.
Joanne Martin pointed out that the ARCO benchmark suite is well-received
by the seismic community and should be seriously considered by PBWG.
     The discussion then turned to compiler-based benchmarks with Tom
Haupt taking the lead.  He distributed the document entitled "High
Performance Fortran, Fortran-D Benchmark Suite" which is available
vi anonymous ftp from minerva.npac.syr.edu.  Tom H. indicated that
the Syracuse compiler benchmarks can test parsers on a collection
of statements from HPF, accept intrinsic functions, and for-all loops.
He stressed that these benchmarks be able to handle hard-coded
message-passing as well as HPF Fortran.  One goal would be to find
instances where HPF is difficult to use.  Tom mentioned that these
codes deal with parallel I/O, validate HPF Fortran, and generate
meaningful statistics.  The current Syracuse benchmark suite contains
over 50 codes and includes many well-known benchmarks such as
LAPACK, NAS, and Genesis benchmarks.  Tom agreed to download this
suite to machine at UT during the workshop (this was done after the
meeting adjourned on March 2 and the files are available in netlib,
send index from pbwg, for more details).  This suite consumes about
5 megabytes and comprises about 14 directories of README files, makefiles,
and source files.
     Brian LaRose (student of Michael B.) then gave a demonstration of
the Performance Database Server (PDS) which is available in an upcoming
release of XNETLIB.  Beforehand, Jack D. gave a brief introduction of XNETLIB 
to familiarize participants with the features of the X Windows-based 
user interface to NETLIB.  The members were pleased with the tool and
made suggestions for improvements such as graphing data retrieved from
the database.  Brian demonstrated the various browsing, rank-ordering, 
and search features of the system he is designing for his Master thesis
at the Univ. of Tennessee.  Michael B. indicated that participants would
be able to test PDS and should send anonymous ftp address/directory information
to "[email-masked]" so a beta release of the software could be sent
to them.
    At this point, Roger Hockney asked that Jack D. define/classify
the publically-available benchmarks that might be considered for the PBWG 
benchmark suite.  The classification into Kernels, Applications, and
Compact Applications is given below.  A (*) indicates availability
as single-node benchmarks also.  Additional benchmarks that were not
explicitly classified are listed under "Miscellaneous".

Kernels:
-------
Linpack 100     (*)
        1000    (*)
Livermore Loops (*)
Los Alamos      (*, some of them)
Tom Dunigan's   (ORNL)
Genesis
Global Operations (e.g., sum, gather/scatter)
NAS
Sorting

Applications:
------------
Perfect
Slalom
RAPS
PEPS

Compact Applications:
--------------------
NAS
Genesis
Syracuse
ARCO
Euroben (*)

Miscellaneous:
------------
flops.c
IOZONE
Dhrystones
Whetstones
Xstones

      Jack D. proposed that the selected codes for the PBWG suite
be implemented in Fortran-77, HPF, in PVM for message-passing 
(this will be converted to MPI, Message Passing Interface, when
the standard is complete).  Roger H. stressed the need to measure
startup latency (alpha, beta), and metrics such
as R-sub-infinity and N-sub-one-half.  Other target measures discussed
included: peak rates, flop/s, communication costs, and bisection
(node-node) bandwidth.  David Bailey stressed that the single processor
benchmarks should be separated from the parallel benchmarks too avoid
confusion.  

     At 5:00pm EDT, the workshop adjourned for a 6:30pm EDT dinner
appointment at Chesapeakes's, a downtown Knoxville seafood restaurant.

Agenda: March 2, 1993
---------------------
    At 9:00 am EDT, Roger Hockney opened the morning session of the
PBWG workshop in Room 221 of the University Center on the campus
of the University of Tennessee.  Prior to any technical discussion on
the makeup of the PBWG benchmark suite.  Participants agreed to hold
a "Birds of a Feather" session at the Supercomputing'93 conference
in Portland, OR in November.   Joanne Martin will schedule this meeting
for the group - a 4:30 pm session was agreed upon by the participants.
Prior to the Supercomputing'93 meeting, Jack D. agreed 
to collect all desirable benchmarks and install them in NETLIB for
public access.  The discussion on the categorical makeup of a
PBWG benchmark suite was then lead by Tony Hey.  The 3 categories
considered were:  Low-Level, Kernels, and Compact Applications.
David W. stressed that 10 would be an optimal number of codes to
use.  Participants generally agreed with this proposal.  Potential
benchmarks considered are listed by category below.  Jack D. stressed
that each group "justify" benchmarks by indicating exactly what
machine feature(s) are to be tested and try to avoid overlap between
categories.

Low-Level:
---------
Start-up, latency, bandwidth
Reduction (broadcast, sum, gather/scatter)
Synchronization (e.g., SYNCH1 from Genesis)
I/O

Kernel:
------
Matrix operations (e.g., multiply, transpose)
LU Decomposition
PDE Solvers (Red/Black Relaxation)
Multigrid
FFT
Conjugate Gradient

Compact Applications:
--------------------
Particle-In-Cell codes (e.g., LPM1 from Genesis)
QCD
Molecular Dynamics
CFD
ARCO
Financial Applications 

      At this point, Roger H. asked that subgroups be formed to
address the each of the above categories of benchmarks.  Each
subgroup will be ask to give a report at the Portland Meeting
concerning the potential design/makeup of their category of
PBWG benchmarks.  Participants agreed that the subgroups be
open to other experts from the performance community: vendors,
academia, and users.  A person may be in more than one subgroup
and Jack D. suggested that a voting strategy be developed.
A "1 vote per institution" was one candidate for consideration but
nothing official was decided at the workshop.  Suggestions for
subgroup members not present at the workshop are included below.
A (*) indicates subgroup leader.

                 PBWG Benchmark Suite Subgroups:
                 ------------------------------

Low-Level           Kernel              Compact Applications     Methodology
---------           ------              --------------------     -----------
PEPS/Warwick        D. Walker              D. Walker (*)          D. Bailey (*)
IBM/D. Frye         J. Dongarra            J. Martin              D. Schneider
R. Pennington       T. Hey (*)             E. Brocklehurst        PEPS
R. Hockney (*)      T. Haupt               K. Ghosh
J. Dongarra         PEPS                   M. Berry
                    E. Kushner (Intel)     D. Bailey
                    D. Bailey
                    P. Wang
                    D. Barton

The following mail reflectors have been setup:

Low Level:            [email-masked]
Kernel:               [email-masked]
Compact Applications: [email-masked]
Methodology:          [email-masked]

Jack D. agreed to set up mail reflectors for each of the subgroups.
(These mail reflectors are setup and functioning.)
All participants agreed to 2 meetings that would be open to anyone
interested in attending prior to the meeting at Supercomputing'93 in
Portland.  Drafts of the reports from each subgroup are due by the
agreed-upon May 24 meeting at the Univ. of Tennessee which will
begin at 8:30 am.  The second meeting date will also be at the Univ.
of Tennessee on August 23-24.  Here's the approved schedule of PBWG meetings:

              Third  Meeting:  May 24, begins at 8:30am (Univ. of Tennessee)
              Fourth Meeting:  August 23-24             (Univ. of Tennessee)
              Fifth  Meeting:  Week of Nov 15  (Supercomputing Portland, OR)

Joanne Martin mentioned that there was a Conference on Performance Tools
on April 1-4 and Tony H. suggested that PBWG might submit an announcement
to that conference about PBWG's activities.  A discussion on possible
name changes for PBWG was considered again.  Some suggestions included
PES(Performance or Parallel Evaluation Suite) and PARKBENCH(Parallel
Kernel Benchmarks).  Since no consensus was reached, the discussion
is postponed to the next meeting so members can think of other names.
In preparation for the May meeting, subgroup leaders briefly mentioned
some of the issues they hope to address:

T. Hey    : talk to vendors, provide list of kernel areas,  justification
D. Walker : define areas, acquire candidate codes, address scalability,
            address problem sizes (e.g., 4 used in Genesis), be able to
            specify data layouts (use vanilla layouts)
R. Hockney: construct list of desirable low-level benchmarks

Participants agreed that users should be able to optimize the suite
by any means provided they document what they do (e.g., assembly
language allowed).  When possible, benchmark contributions should
include the code that was used to obtain the reported results (stored
in PDS).  
     The second day of the PBWG workshop was adjourned by Roger Hockney
at 11:00am EDT, and participants had informal discussions before
departing the Univ. of Tennessee campus.  PBWG activities will
be posted to the comp.parallel newsgroup (USENET).