Subject: Minutes of May ParkBench Meeting

From: "Michael W. Berry" <[email-masked]>
Date: Fri, 23 May 1997 11:05:31 -0400

Here are the minutes from the recent ParkBench meeting in Knoxville.
Best regards,
Mike

-----------------------------------------------------------------
Minutes of ParkBench Meeting - Knoxville Hilton, May 9, 1997
-----------------------------------------------------------------

ParkBench Attendee List:

     (MBa) Mark Baker          Univ. of Portsmouth   [email-masked]
     (MBe) Michael Berry       Univ. of Tennessee    [email-masked]
           Shirley Browne      Univ. of Tennessee    [email-masked]
     (JD)  Jack Dongarra       Univ. of Tenn./ORNL   [email-masked]
           Jeff Durachta       Army Res. Lab MSRC    [email-masked]
     (VG)  Vladimir Getov      Univ. of Westminister [email-masked]
     (CG)  Charles Grassl      SGI/Cray              [email-masked]
     (TH)  Tony Hey            Univ. of Southampton  [email-masked]
     (AH)  Adolfy Hoisie       Los Alamos Nat'l Lab  [email-masked]
     (CK)  Charles Koelbel     Rice University       [email-masked]
     (PM)  Phil Mucci          Univ. of Tennessee    [email-masked]
           Erik Riedel         GENIAS Software GmbH  [email-masked]
     (SS)  Subhash Saini       NASA Ames             [email-masked]
     (RS)  Ron Sercely         HP-Convex             [email-masked]
           Alan Stagg          CEWES                 [email-masked]
     (ES)  Erich Strohmaier    Univ. of Tennessee    [email-masked]
     (PW)  Pat Worley          Oak Ridge Nat'l Lab   [email-masked]

SPEC-HPG Visitors:

           Don Dossa           DEC                   [email-masked]
     (RE)  Rudi Eigenmann      Purdue University     [email-masked]
           Greg Gaertner       DEC                   [email-masked]
           Jean Suplick        HP                    [email-masked]
           Joe Throp           Kuck & Associates     [email-masked]

At 9:05am EST, TH opened the meeting and ask that all the attendees
introduce themselves.  After a brief overview of the proposed agenda,
MBe reviewed the minutes from the last ParkBench meeting in October
of '96.  The minutes were unanimously accepted and TH asked VG to
present the proposed changes to the low-level benchmarks (9:20am).

VG reviewed the original COMMS1 (ping-pong or simplex communication) and
the COMMS2 (duplex communication) low-level benchmarks.  He discussed
some of the problems with the previous versions.  These included the
omission of calculated bandwidth, large message length problems, and
large errors in the asymptotic fit.   In collaboration with RS and CG,
a number of improvements have been made to these benchmarks:

	1. Measured bandwidth is provided in output.
	2. Time for shortest message is provided.
	3. Maximum measured bandwidth and the corresponding message
	   length is now provided.
	4. The accuracy of the least-squares 2-parameter fit has been
	   improved (sum of squares of the "relative" and not absolute
	   error is now used).
	5. New 3-parameter variable-power fit for certain cases added.
	6. Can report parametric fits if the error is less than some
   	   user-specified tolerance.
	7. Introduce KDIAG parameter to invoke diagnostic outputs.
	8. Modifications fo ESTCOM.f (as suggested by RS).
    
CG pointed out that it may not always be possible to interpret zero-length 
messages for these codes.  On the Cray machines, such messages force an 
immediate return (i.e., no synchronization).  He proposed that allowing zero-
length messages be removed for the COMMS benchmarks.  RS showed an actual
COMMS1 performance graph demonstrating the difficulty of data extrapolation
(if used to get latency for zero-length message-passing).  RS pointed out,
however, that zero-length message are defined w/in MPI, and suggested that
a simple return (as in the case of Cray machines) is not standard.

VG displayed some of the observed COMMS1/2 performance obtained on the
Cray T3E.  The 3-parameter fit yielded a 7% relative error for messages
ranging from 8 to 1.E+7 bytes.  CG questioned how the breakpoints were
determined?  He indicated the input parameters to the program required
previous knowledge of where breakpoints occur (although implementations
could change constantly).  TH suggested that the parametric fitting should
not be the default for these benchmarks, i.e., separate the analysis from
the actual benchmarking (this concept was seconded by CG).  RS suggested
that the fitting routines could be placed on the WWW/Internet and the
COMMS1/2 codes simply produce data.  CK, however, stressed that the codes
should maintain some minimal parametric fitting for clarity and
consistency of output interpretations.  

The minimal message length shown for the T3E results shown by VG was 8 and
the corresponding minimal message length for a Convex CXD set of
COMMS benchmarks was 1.  The lack of similar ranges of messages could
pose problems for comparisons.  JD strongly felt that users will return
to the notion of "latency" and want zero-length message overheads.  Users
may be primarily interested in start-up time for message-passing.  RS pointed
out that MPI does process zero-length messages.  JD suggested that
the minimal message length for the COMMS benchmarks be 8 bytes and RS proposed
that the minimal message-passing time and corresp. message length be
an output.  After more discussion, the following COMMS changes/outputs were 
unanimously agreed upon:

	1.  Maximum bandwidth with corresp. message size.
	2.  Minimum message-passing time with corresp. message size.
	3.  Time for minimum message length (could be 0, 1, 8, or 32 bytes
            but must be specified).
	4.  The software will be split into two program: one to report
	    the spot measurements and the other for the analysis.

At 10:00 am, SPEC-HPG members joined the ParkBench meeting for a joint
session.  CK reviewed the DoD Modernization Program.  He indicated that
the program is based on 3 primary components:

	1. CHSSI (Commonly Highly Scalable Software Initiative)
	2. DREN (Defense Research & Engineering Network)
	3. Shared Resource Centers (4 Major Shared Resource Centers or
           MSRC's and 20 Distributed Centers or DC's)

Benchmarking is part of the mission of the MSRC's, especially for
system integration and the Programming Environment & Training (PET)
team.  CK mentioned that the resources available at the MSRC's include:

256-proc. Cray T3E, SGI Power Challenge (CEWES), 256 proc. IBM SP/2 and
SGI Origin 2000 at ASC, SGI 790 at NAVO, and a collection of {SGI Origin,
Cray Titan, J90} at the Army Research Lab.

The benchmarking needs of the DoD program can be categorized as either
contractual or training.  The contractual needs are specified as PL1
(evaluation of initial machines), PL2 (upgrade to gain 3 times the
performance of PL1), and PL3 (upgrade to gain 10 times the performance
of PL1).  CK mentioned that the MSRC's are planning for the PL2 phase
later this year with PL3 scheduled in approx. 3 years.
The training needs include: the evaluation of programming paradigms,
the evaluation of performance trade-offs, templates for designing new
codes, and benchmarks for training examples.

The contractual benchmarks comprise 30 benchmarks (22 programs) some
of which are export-controlled or proprietary (data may not be used
in the public domain in some cases).  The run rules specify the number
of iterations for each benchmark in the suite.  Each MSRC uses a different
number of iterations per benchmark.  Code modifications are allowed (parallel
directives and message-passing can be used but no assembler) and algorithm
substitutions are permitted provided the problem does not become specialized.
The only performance metric reported for these benchmarks is the elapsed
time for the entire suite.  Benchmarks can be upgraded to reflect current
workloads of the MSRCs but they must be compared head-to-head with 
previous systems.

Example codes included in the DoD benchmark suite include: CTH (finite
volume shock simulation), X3D (explicit finite element code), OCEAN-O2 (an
ocean modeling code), NIKE3D (implicit nonlinear 3D FEM), and Aggregate
I/O benchmark.

Planned benchmarking activites for the DoD Modernization Program include:

	1. benchmarks for evaluating programming techniques (determine what
           works; develop decision trees)
	2. benchmarks for teaching (classes on "worked" examples; template
           modification)

This effort currently has 1 FTE and over 50 University personnel (in PET
program) involved (although they are not primarily responsible for
benchmarking work).

At 10:35am, TH asked AH from Los Alamos Nat'l Lab to overview their ASCI
benchmark suite.  He began by pointing out that these codes formulate the
"Los Alamos set of" ASCI Benchmarks.  Before presenting the list of codes,
AH noted that the philosophy of this activity was to achieve 
"experiment ahead" capability especially with immature computing platforms.
Los Alamos is also interested in developing performance modes as well as
kernels.  The list of active/research codes and compact applications 
comprising this suite are:

Code		Language(s)	Parallelism 	Description           
    
*HEAT(RAGE)	f77, f90	MPI(f90)	Eulerian adaptive mesh
				MPIfSM(f77)	refinement based on
						Riemann solvers; coupled
						physics-CFD; particle &
						radiative transport

EULER		f90		MPI		Admissable fluid (for SIMD);
				SIMD(SP		unstructured mesh, explicit
				vector)		solution; high-speed fluids;
						SP=single processor

NEUT		f77		MPI,SM,		Monte-Carlo, particle
				SHMEM

SWEEP3D		f90		MPI, SHMEM	Inner/outer iteration (kernel)
                                                (compact application)

HYDRO(T)	f77		Serial          (compact application)

TBON		f77		MPI		Material science; quantum
						mechanics; polymer age    
						simulation

*TECOLOTE	C++		MPI		Mixed call hydro. with regular
						structured grid

*TELURIDE	f90		MPI		Casting simulation; irregular
						structured grid; Krylov solution
						methods

*DANTE		HPF		MPI

* = export controlled

The codes and compact apps above vary in size from 2,000 to 35,000 lines.
AK noted that LANL could provide support for future ASCI-based ParkBench codes. 
The ASCI benchmark suite presented might include in the future tri-lab
(Livermore, Sandia, Los Alamos) contributions.  The ASCI application suite can
be set up with data sets leading to varying run-times.  AH mentioned that Los 
Alamos' ASCI benchmarking efforts are focused on high performance computing,
leading edge architectures, algorithms, and applications.  They are 
particularly concentrating in developing expertise in distributed shared-memory
performance evaluation and modeling.  AH expressed the hope that the efforts of
ParkBench will follow similar directions.

At 11:05am, SS reviewed some of the most recent NAS Parallel Benchmarks results.
He began with vendor-optimized CG Class B results using row and column 
distribution blocking.  Results for different numbers of processors of the T3D 
were reported along with results for the NEC SX-4, SGI Origin 2K, Convex SPP2K,
Fujitsu VPP700, and IBM P2SC.  He also showed results for FT Class B and BT 
Class B (all machines reported performed well on this benchmark).  For BT, it 
was pointed out that 4 of the machines (Cray T3E, DEC Alpha, IBM P2SC, and NEC 
SX-4) essentially are based on the same processor but achieve widely-varying
results.  SS also reported HPF Class A MG results on 16 processors of the IBM 
SP2.  The HPF version (APR-HPF/Portland Group compiled) was only 3 times slower
than the MPI-based (f77) implementation.  This is indeed a significant result 
given that two years ago the HPF version was as much as 10 times slower than 
the comparable MPI version.  An HPF version of the Class A FT benchmark on 64 
processors was shown to be faster than the MPI version (1.6 times faster) when
optimized libraries are used in both versions.  For the Class A SP benchmark
(on 64 processors of the SP/2), the APR- and PGI-compiled HPF versions were 
within a factor of 2 of the MPI versions.  Finally, the HPF Class A BT code on 
64 processors of the Cray T3D was within a factor of 0.5 of the MPI version.

At 11:35am, TH invited RE to overview current SPEC-HPG activities.  The SPEC-HPG
benchmarks define a suite of real-world high-performance computing applications
designed for comparisons across different platforms (serial and message-
passing).  RE pointed out the history of the SPEC-HPG effort as a merger between
the PERFECT and SPEC benchmarking activities.  The current SPEC-HPG suite is
comprised of 2 codes: SPECchem96 and SPECseis96.  The SPECchem96 code evolved
from the GAMES code used in pharmaceutical and chemical industries.  It
comprises 109,389 lines of f77 (21% comments), 865 subroutines and
functions.  The wave functions are written to disk.  The SPECseis96 code
is derived from the ARCO benchmark suite which consists of four phases: data
generation, stack data, time migration, and depth migration.  This code
decomposes the domain into n equal parts (for n processors) with each part
processed independently.  It is have over 15K lines of code made up of
230 Fortran subroutines and 199 C functions for I/O and systems utilities.
SPECseis96 uses 32-bit precision, FFT's, Kirchoff integrals, and finite
differences.

The very first set of SPEC-HPG benchmark results were approved on May 8,
1997 (preceding day).  New benchmarks being considered are PMD (Parallel
Molecular Dynamics) and MM5 (NCAR Weather Processing C code).  The decision
on whether or not to accept these two potential SPEC-HPG codes will be made
in about 5 months.  The SPEC-HPG run rules permit the use of compiler
switches, source code changes, optimized libraries (which have been
disclosed to customers).  Only approved algorithmic changes will be disclosed.
RE gave the URL for the SPEC-HPG effort: http://www.specbench.org/hpg.  He
also referred to a recent article by himself and S. Hassanzadeh in "IEEE
Computational Science & Engineering" and two email reflectors for SPEC-HPG
communication: [email-masked] and [email-masked].

JD then gave a brief history of ParkBench and SPEC-HPG interactions and
suggested that the two efforts might consider sharing results and software.
The biggest difference in the two efforts is in the availability of
software as ParkBench code is freely available and SPEC-HPG software
has some restrictions.  A forum to publish both sets of results was discussed
and it was agreed that both efforts should at least share links on their
respective webpages.  RE pointed out that anyone can get the SPEC-HPG CD
of benchmarks without actually being a SPEC member.

JD stressed that the process of running codes (for any suite) needs to
be simplified so that building executables for different platforms is not
problematic.  Modifications for porting should be restricted to driver programs.
RS indicated that he has Perl scripts that runs all low_level, including 
COMMS3 for 2 to N procs, and produces a summary of the results. 

*** ACTION ITEM ***
JD, RE, AH, and CK will discuss a potential joint effort to simplify the
running of benchmark codes (contact RS also about his Perl scripts).

MBa noted that the SPEC-HPG members should be added to the ParkBench
email list ([email-masked]).  He also indicated that European
benchmarking workshop scheduled next Fall might coordinate with the
European SPEC group (scheduled for Sept. 11-12).

At 12:10pm, the attendees went to the lunch (Soup Kitchen).

After lunch (1:30pm), TH asked ES and VG to coordinate changes to the
COMMS benchmarks discussed above (*** ACTION ITEM ***).  ES then discussed
modifications to poly2 for the ParkBench V2.2 suite.  The proposed changes
include
	1. enlarged arrays A(1000000), B(1000000)
	2. removal of arrays C and D
	3. avoid cache flush (use a sliding vector), i.e., 

             DO I=1,N               DO I=NMIN,NMAX
                         becomes       ...

                                       NMIN=NMIN+N+INC

           where INC=17 by default (avoids reuse of the old cache line).

PM then discussed a program for determining parameters for memory subsystems.
Characteristics of this software include the use of tight loops, independent
memory references, maximized register use.  He showed graphs of memory
hierarchy bandwidth (reads and writes) depicting memory size (ranging from 4Kb
to 4Mb) versus Mb/sec transfer rates.  Some curves illustrated the effective 
cache size quite well.  PM pointed out that dynamically-scheduled processors
pose a significant problem for this type of modeling.  The program can be
run with or without a calibration loop exploiting known memory transfer data.
CG suggested that it would be nice to have such a program to measure latency
at all levels of the hierarchy.  PM's webpages for this program are:

	http://www.cs.utk.edu/~mucci/cachebench and
	http://www.cs.utk.edu/~mucci/parkbench.

CK suggested that an uncalibrated version of PM's benchmark would be more
useful to users (more reflective of real codes).  JD pointed out that the
output of the program could be tabulated bandwidths, latencies, etc.  CG
felt this program would be a very useful tool.  PM noted that the calibration
will not be used by default.  TH suggested that the ParkBench effort might
want to develop a future "ParkBench Tool Set" which contains progams like
this one developed by PM.

With regard to the Linalg Kernels, ES noted that although many of the
routines have calls to Scalapack routines, Scalapack will not be included
in future software releases.  Users will have to ge their own copies of
the source (or binaries) for Scalapack.  The size of these particular
kernel benchmarks drops by a factor of one-third by removing Scalapack.

*** ACTION ITEM ***
ES will report the most recent Linalg benchmark performance results at the
next ParkBench meeting.

TH then asked for discussions on new benchmarks with MBa leading the
discussion on HPF benchmarks.  MBa indicated that a new mail reflector
([email-masked]) had been set up for this cause with himself
as moderator for low-level codes (CK will moderate kernels and SS will
moderate discussions on HPF compact applications).  MBa noted that there
is limited manpower for the HPF benchmarking activities.  CK noted that
he had discussed this effort at recent the HPFF meeting (and other
users meetings).  A draft document on the ParkBench HPF benchmarks is
available at http://www.sis.port.ac.uk/~mab/ParkBench.  MBa felt strongly
that without manpower support this particular activity will die and that
a lead site is needed.

*** ACTION ITEM ***
CK and SS will investigate interest in HPF compact application development.

JD indicated that wrappers are being used to create HPF versions of the
Linalg kernels.  The procedure involves writing wrappers for the current
Scalapack driver programs.  Eventually, these programs may be completely
rewritten in HPF (this will start in the summer).  TH suggested that HPF
kernel benchmark performance be reported at the ParkBench meeting 
in September (at Southampton Performance Workshop).

MBa went on to report on the status of I/O benchmarks.  Basically, not
much progress has been made on the ParkBench I/O initiative.  A new I/O 
project between ECMWF, FECIT, and the Univ. of Southampton was launched
this past February.  They are looking at the I/O  in the IFS code from
the ECMWF (European Weather Forecasting).  David Snelling is the FECIT
leader who has also participated in ParkBench activities.  This I/O
project has 1 FTE at Southampton and 1.5 FTE at FECIT along with several
personnel at ECMWF.  One workshop, two technical meetings for the 1-year
project is planned.  The goals are: to develop instrumented I/O
benchmarks and build on top of MPI-IO (test, characterize parallel
systems).  Their methodology is very similar to that of ParkBench.
Codes in f90 and ANSI C are being considered (stubs for VAMPIR and
PABLO).  Regular reports to Fujitsu (sponsor of activity) are planned
and a full I/O test suite is planned by February 1998.

MBa also reported on the status of the ParkBench graphical database.
Currently, the performance data is kept in a relational DBMS.  A
frontend Java applet has been written to query the DBMS on-the-fly.
A backend is also in development which will automate the extraction
of new performance data and insertion into the DBMS (via an http
server).  By September, a more complete prototype which will allow
MS access and JDBC between 2 different machines should be ready.

VG then discussed the development of Java-based low-level benchmarks.
He presented a Java-to-C Interface Generator which would allow Java
benchmarks to call existing C libraries on remote machines.  He
presented sample Java+C NAS PB results on a 16-processor IBM SP/2
(Class A IS Benchmark):

           Version        1 Proc  2Procs  4 Procs  8 Procs  16 Procs
           NASA (C)        29.1    17.4     9.4     5.2        2.8
                 C         40.5    24.9     13.1    9.3       15.6
              Java         ----   132.5     64.7   37.9       33.5

At 2:50pm, TH reported other ParkBench activities including the
new PEMCS (Performance Evaluation and Modeling for Computer Systems)
electronic journal.  Suggested articles/authors include:

       *1. ParkBench Report No. 2 (ES, MBe)
       *2. NAS PB
	3. SPEC-HPG
       *4. Top 500
	5. AutoBench (M. Ginsburg)
       *6. Euroben (van der Steen)
	7. RAPS
	8. Europort
       *9. Cache benchmarks
       10. ASCI benchmarks (DoD)
      *11. PERFORM
       12. R. Hockney
      *13. PEPS
       14. C3I/Rome Labs

Those articles possible for Summer '97 are marked via *.  JD suggested
that articles be available in Encapsulated Postscript, PDF (Adobe),
and HTML.  TH noted that EU funding will provide a host computer and
some administration.  Possible publishers are Oxford Univ. Press and
Elsevier.

At 3:10pm, ES requested more items for the ParkBench bibliography
which will be available on the WWW.  PW suggested that authors should
be able to submit links to ParkBench-related applications.  JD then
briefly discussed WebBench which is a website focused on benchmarking
and performance evaluation.  Data is presented on platform,s applications,
organizations, vendors, conferences, papers, newsgroups, FAQ's, and
repositories (PDS, Top500, Linpack, etc.).  The WebBench URL is
http://www.netlib.org/benchweb.

MBa reminded attendees of the Fall Performance Workshop/ParkBench
meeting on (Thursday and Friday) Sept. 11 and 12.  This meeting
will be held at Venue, County Hotel, Southampton, UK.  Invited
and contributed talks will be presented.

With regard to ParkBench funding, JD indicated that the UT/ORNL/NASA
Ames proposal was not selected for funding but that it could be re-
submitted next year.  Expected funding from Rome lab was not received.
TH and VG did not succeed this past year either although some funding
from Fujitsu is possible.

TH adjourned the meeting at 3:25pm EST.