Subject: Revised Minutes

From: "Michael W. Berry" <[email-masked]>
Date: Tue, 07 May 1996 13:24:51 -0400

A few more changes... Mike

-----------------------------------------------------------------
Minutes of Parkbench Meeting - Knoxville Hilton, April 26, 1996
-----------------------------------------------------------------
Attendee List:

     (MB)  Michael Berry       Univ. of Tennessee    [email-masked]
     (JD)  Jack Dongarra       Univ. of Tenn./ORNL   [email-masked]
     (VG)  Vladimir Getov      Univ. of Westminister [email-masked]
     (MG)  Myron Ginsberg      EDS/General Motors    [email-masked]
     (TH)  Tony Hey            Univ. of Southampton  [email-masked]
           Edgar Kalns         IBM                   [email-masked]
           David MacKay        Intel SSD             [email-masked]
     (RM)  Richard C. Metzger  RL/CJCB (Rome Lab)    [email-masked]
           Phil Mucci          Univ. of Tennessee    [email-masked]
     (BP)  Bodo Parady         Sun Microsystems      [email-masked]
     (SS)  Subhash Saini       NASA Ames             [email-masked]
     (ES)  Erich Strohmaier    Univ. of Tennessee    [email-masked]
     (PW)  Pat Worley          Oak Ridge Nat'l Lab   [email-masked]

[Events after 11:30am]

Regarding the Low-Level benchmarks:
-----------------------------------
SS showed that variations ranging from 18-30% on an IBM SP/2 have
been recorded for the low-level codes.  There is a proceedings
paper (SC'94) which first noted this.  Members felt the PVM
daemons may be the suspects here.  TH reported that strange results
had been obtained for COMMS1 with respect to the MAXLEN parameter and
that the MPI and PVM versions of COMMS2,3 may not measure the same
operations.  TH demonstrated performance variations of COMMS1,2 on the
IBM SP/2 and Meiko CS2.  The best results were obtained using PSEND and 
PRECEIVE on contiguous data.  It was noted that PSEND/PRECEIVE were
not available with the original COMMS1,2 codes were written.  TH noted
that memory access should be measured as well and that perhaps COMMS3
could be rewritten for consistency with the current COMMS1,2 codes (ES
suggested this could be done for the V3.0 Parkbench Release).

VG suggested that the group rethink the goals of low-level benchmarking.
He proposed 3 goals of this level of benchmarking:

(1) Comparisons across different parallel computers,
(2) Performance estimation of parallel kernels and applications, and
(3) Performance tuning of parallel applications.

For single node architectures, VG suggested that scalar performance,
vectorization, and memory hierarchies be measured.  He pointed out that
RINF1 which comprises 17 examples of code fragments/kernels is 15 years
old and was best suited for vector pipeline machines.  Therefore, a new
low-level benchmark was proposed for evaluation of the memory hierarchy
effects.  VG also suggested that vendors be allowed to use their single
node sustainable performance kernels (in-house versions) for such
benchmarks. Members felt that a new benchmark of memory hierarchies would
be very useful in addition to the existing low-level codes.

At 12:30pm EDT the group went to lunch at the "Soup Kitchen" in downtown
Knoxville.

At 1:20pm EDT, the meeting resumed.  MB and TH briefly reviewed the Minutes
of the November 11, 1995 Parkbench Meeting (in Knoxville).  JD pointed out
that Cray Research should be using the BLACS provided by Parkbench to avoid
the problems observed with task spawning on the Cray T3D.  TH and JD indicated
that they will investigate the use of SGI machines since little or no results
have been obtained on those machines to date.  BP announced that SPEC-HPSC
will release their first benchmark suite in the Summer of '96 and will have
two application codes (SEISMIC (ARCO) and  CHEMISTRY).

The meeting focus then switched to Compact Applications:
--------------------------------------------------------

MG discussed the status and needs of benchmarking in the automotive
industry.  USCAR released a benchmark suite at the end of August 1995.
They typically use 5-6 commercial codes for benchmarks based on older
model cars.  They have had discussions with Europort and seek their
models as well.  The particular needs of benchmarking in this area include:

(1) Information on the limitation of machines,
(2) Crossover points between small-medium-large machines - would like
    to see results posted on the WWW someday,
(3) Assess if machine is general purpose or only good for a specific model,
(4) Develop a canonical set of parameters for scalability analysis,
(5) Develop application-dependent metrics, and
(6) Focus on shared-memory paradigms (message-passing not desirable).

Ford has a lot of Cray C-90's and uses them primarily for job streaming.
Most auto companies do not do parallelism and depend on commercial 
Finite Element Model (FEM) codes.  Typical machines benchmarked include:
IBM SP/2, Convex Exemplar, Cray J9-16, and SGI PowerChallenge.  Five
commonly-used commerical benchmark codes (which are also in the Europort
suite) are: MSC/NASTRAN, STARCD, ABACUS, LSDYNA, and PAMCRASH.  MG pointed
out that parallel commercial FEM codes have not evolved (few number of
processors typically sufficient).  Measurements based on turn-around-time
are needed.  Assessing the performance of job mixes with increasing numbers
of threads is very desirable (throughput).  Creating a synthetic job mix
is more desirable than running one code on a single dedicated machine.  Four
areas MG suggested for future benchmarks are:

(1) Crashworthiness (large structures, 60K degrees of freedom),
(2) Other structures problems (noise vibration, heat),
(3) CFD (exterior aerodynamic, internal combustion),
(4) Manufacturing processes (metal forming).

MG concluded his presentation with a note that a preliminary report
on AUTOBENCH was available (copy left with TH/JD).

SS then presented some results on HPF-based NAS Parallel Benchmarks (PB).
He pointed out the heavy investment at NAS in CMF on the CM-2 and CM-5
(shutdown on March 1995).  He showed that for the FT
benchmarks, that the MPI versions were about 2 times faster than the
MPI versions of NAS PB Version 2.0 on the IBM SP/2.  Also
he showed that for the SP benchmark, that the MPI version was about 3
times faster than the MPI versions of NAS PB Version 2.0 on the IBM SP/2.
For the BT benchmark, the MPI code was 4 times faster than the HPF counterparts. 
The Portland Group HPF compiler was used on the IBM SP/2 tested.  He alluded
that the gain in using MPI may not always be cost-effective given the man
hours required to obtain such codes.  With HPF, once can get a speedup factor
of 2 over baseline (16 processor) versions for 40+ processors on the
IBM SP/2.  Having a library of HPF/MPI codes is a big win given the
ease of programming with HPF and good performance of underlying MPI
kernels.  SS also pointed out that a HPF-based PIC (Particle-In-Cell)
code had obtained a speedup of 5 on a 16 processor SGI Power Challenge
XL.  SS concluded his presentation by indicating that

(1) CMF-to-HPF conversion is easy,
(2) Lack of parallel I/O in HPF is a definite problem,
(3) Lack of parallel math and scientific libraries in HPF will
    complicate the CMF-to-HPF conversion (and HPF ports), and
(4) HPF compilers are maturing. 

TH suggested that HPF versions of the Parkbench kernels and compact
applications be considered for Release Version 3 of the Parkbench Suite.

RM from Rome Lab then proceeded to describe the C3I Parallel Benchmark
Suite.  He noted that DARPA is supporting efforts in real-time parallel
benchmark development (configurable computing).  The C3I compact
application level codes are just over 1K lines each and can be accessed
on the WWW at http://www.se.rl.af.mil:8001.  Applications involve SAR
processing, multiple hypothesis tracking, and terrain meshing.  Such
codes typically scale down problems to meet constraints with smaller numbers
of processors (not necessarily real-time).

The real-time benchmark effort is just starting at Rome Lab with 
an effort mimicking the Parkbench effort (kernels and compact applications).
RM proposed a future "real-time annex" to Parkbench.  He noted that both
functional and temporal specifications will undoubtedly result in new
performance metrics to consider.  JD indicated his willingness to extract
2 of these codes and integrate them into the next Parkbench release.

PW of ORNL then discussed some of the performance results obtained for
the shallow water compact application code on an Intel Paragon (up to
512 processors).  He indicated that for up to 128 processors, the percentage
of time due to communication is not dominating.  For over 256 processors,
the communication can consume as much as 30% for large problems and 60%
of the total execution time for small problems.  Scaling up problems sizes
makes no sense for weather-related codes (i.e., problem sizes are fixed).
PW also mentioned a MPI protocol sensitivity study with emphasis on
the collective communication performance of MPI.  He compared MPI
communication routines versus hand-coded Fortran with optimal NX protocols
and showed that the handcoded was consistently better, indicating that, in
particular, there is room for performance improvement in the MPI collective
routines. He also mentioned that similar results hold on the IBM SP2 and on
the Cray T3D. 

TH then listed the compact applications of current and future interest
for Parkbench:

1. CFD/Aero  (NAS PB)
2. Weather/Climate  (Shallow Water, POLMP/IFS)
3. Seismic (ARCO)
4. Autobench  (AVL/Fire)
5. C3I (Non Real-Time)
6. C3I (Real-Time)
7. Particle-In-Cell (PIC) code
8. FDMOD

At 3:10pm EDT, TH mentioned that the Electronic Benchmarking Journal
now has support and suggested that articles on C3I, Autobench, Parkbench,
and NAS PB be submitted.  It was suggested to wait for Parkbench Release
2.0 to kickoff the Electronic Journal.

*** Further Actions ***

I.   Release 2.X + Results
     Low-level (COMMS2,3)
     Put results in standard format such as that used in NAS PB.
     Run-rules
     Add NAS PV Version 2.2 results (SS will provide).
     Get SGI results (TH and JD may contact Tony S. at Miss. State).
     Mirror PDS/GBIS at both US and UK sites.

II.  Discussion on Release 3 (2.x)
     Codes at all levels.
     TH will provide POLMP.
     BP can provide FDMOD documentation (85% seismic).
     Roger Hockney will be asked for a PIC code.

III. The next Parkbench Meeting is scheduled for late August/early September 
     1996 in Knoxville.

TH adjourned the meeting at 3:23pm EDT.

---

From: [email-masked] (Rick Metzger)
Date: Thu, 9 May 1996 21:00:51 -0400

Sorry this is late but my organization is RL/C3CB not RL/CJCB.  I
apologize.  The nuns in school always said I had bad handwriting!

Thanks,
Rick

>A few more changes... Mike

*****************************************************************************
Richard C. Metzger                              Mail Address:
Software for High Performance Computing         Richard C. Metzger
Rome Laboratory - USAF                          Rome Laboratory/C3CB
                                                525 Brooks Rd.
email:                                          Rome, NY 13441-4505
[email-masked]
[email-masked]

voice (315)330-7652
fax   (315)330-7989