Subject: Minutes of last meeting

From: "Michael W. Berry" <[email-masked]>
Date: Thu, 01 Sep 1994 14:51:36 -0400

Please send corrections to the minutes below.
Thanks, Mike

---------------------------------------------------------------------
Minutes of the Parkbench Meeting  - Knoxville Hilton - August 29,1994
---------------------------------------------------------------------

List of Attendees:

Michael Berry   (Univ. of Tennessee, [email-masked])
David  Bailey   (NASA, [email-masked])
Jack Dongarra   (Univ. of Tennessee / ORNL, [email-masked])
Alistair Dunlop (Univ. of Southampton, [email-masked])
Charles Grassl  (CRI, [email-masked])
Roger Hockney   (Univ. of Southampton, [email-masked])
Todd Letsche    (Univ. of Tennessee, [email-masked])
Bodo Parady     (Sun Microsystems, [email-masked])
Fiona Sim       (IBM Kingston, [email-masked])
David Walker    (ORNL, [email-masked])
Pat Worley      (ORNL, [email-masked])

At 9:10am Jack D. provided details of the NSE (National Software Exchange)
on the world-wide-web which will put a framework around high performance
computing (HPC) activities.   The URL is http://www.netlib.org/nse/home.html.

Roger H. started the formal meeting at 9:15am.  Mike B. read the minutes
from the March 28 Parkbench meeting which were accepted with minor
revisions. 

At 9:30am, Roger H. noted that the first Parkbench report has now
appeared in the Summer 1994 (45 pages) issue of "Scientific Programming"
published by Wiley.  Roger noted that minor editing to the current on-line
version was required so the two versions are not quite the same.
The question of reprints was brought up.  Jack D. indicated that a postscript
and HTML version of paper are available and that a reference to hardcopy
could be easily added.  One suggestion was to scan the cover page of the
hardcopy for the HTML version.

Roger H. then asked Fiona Sims to comment on the Parkbench/SPEC-HPSC
relationship.  Fiona gave a brief talk discussion the current interactions
between the two groups.  She provided a list of the
current SPEC/HPSC members and noted that an organization must pay $5K 
to be an Assoc. Member and $1K to be an Affiliate Member.  SPEC's aim is
to look at "real-world" applications for standardized cross platform 
comparison on a level playing field.  The focus is on symmetric multi-
processor systems, workstation clusters, distributed-memory parallel
systems, vector and vector parallel supercomputers.  Fiona stressed that
"end users" are to be ultimate beneficiaries of the SPEC/HPSC efforts.
Basic guidelines have been adopted and their codes must be written in
Fortran 77, ANSI C and MPI or PVM.  They are currently looking at codes
from the following application areas:
seismic data processing (ARCO), computational chemistry (GAMESS & AMBER),
and computational fluid dynamics (TURB3D).  The SPEC/HPSC contacts are
Siamak Hassanzadeh and Fiona Sims.  From a previous conference call involving
Siamak H., Jack D., Fiona S., and David W. it was generally decided that
SPEC/HPSC should draw on Parkbench experience and expertise.  Parkbench is
clearly more research oriented than SPEC/HPSC but Parkbench can draw
draw on SPEC/HPSC experience (commercial user interests, vendor interests).
Fiona encouraged collaboration and interaction between the two groups and
mentioned that a join BOFS at Supercomputing '94 be held.  Another consideration
voiced was to have one join meeting per year (perhaps in Knoxville).
Parkbench participation at SPEC/HPSC meetings was encouraged and the next
SPEC/HPSC meeting is scheduled in September.  SPEC/HPSC lans to have 4 meetings
each year.  Bodo P. pointed out that SPEC is based on vendor compliance 
and is truly a volunteer effort.

SPEC/HPSC has need for application codes based on finite elements and
engineering analysis.  A weather code is being considered but is not a MP
(message-passing) code.  Pat W. mentioned that many versions of ONRL's
Shallow Water code are available (but it is really a compact application).
A human genome application is another possibility.  Fiona S. finished
her discussion at 9:52am.

Jack D. felt that joint meetings at Supercomputing conferences would be
a good idea and that a Parkbench representative would need to attend the
other regularly-scheduled SPEC/HPSC meetings. Fiona S. pointed out that
SPEC/HPSC only wants application codes no kernels.  Jack D. asked where the
funds from SPEC annual fees actually go.  Bodo P. pointed out that some
funds could go to travel costs for affiliate members (university folks).
Both Fiona S. and Bodo P. felt that that universities would need to
formally join SPEC to be able to ask for funds (travel etc.) in order to
build the SPEC HPSC/Parkbench alliance.  Jack D. then indicated that he
would pay $1K to SPEC for Parkbench to be an affiliate member.  Parkbench
would be the member name.  All attendees agreed that Jack D.'s offer, 
which would to allow Parkbench to formally join SPEC/HPSC, be accepted.

ACTION ITEM: Jack D. will contact SPEC/HPSC about Parkbench membership.

At 10:08am, Roger asked for status reports from the subgroups.  David B.
indicated that there had been no activity from the Methodology group.
However, David B. asked if vendors should be asked to conform to certain
code standards?  Would vendors still write low-level codes for absolute
performance.  David W. suggested that for CA's we need sequential, MP, 
and HPF version (CA = Compact Application).  David B. suggested that
separate reporting categories be used (pencil-paper versions allows a
variety of versions to be produced).  Roger H. pointed out that the 
hierarchical structure of Parkbench has had some recent successes 
with FFT's and Genesis Benchmarks.  One can categorize kernels with 3
hardware parameters, for example.  Charles G. pointed out that we need
to get away from single point performance characterization.  David B.
suggested that parameterization of kernels and CA's be on-going research.
Alistair D. pointed out that this can only be done with codes that are
well-understood.  Pat W. suggested that there might be specification
problems with using low-level benchmarks to explain many versions of a
parallel application code.  Fiona S. suggested that it would be good to
instrument/parameterize the CA's.  Roger H. suggested that perhaps a
working group be formed to address the parameterization issue.  Alistair D.
suggested that SOTON's FFT-based results be distributed (FFT code from Genesis)
to the Parkbench members.

Roger H. then reported on the cleanup of the low-level kernels to make 
the Parkbench suite and the Genesis versions the same.  Jack D. will provide
the number of requests for Parkbench codes to date.  It was noted that
most of the codes are written in Fortran 77 and use PVM 3.2.

ACTION ITEM: Jack D. to provide data on Parkbench code requests from Netlib.

Alistair D. then reported on Kernel subgroup effort.  Jack D. indicated that 
the linear algebra kernel components in PVM are now available: dense
matrix multiplication, transpose, dense LU, dense QR, and matrix tri-
diagonalization.  He also noted that results for LU, QR, and tridiagonalization
on the Intel Paragon are also available.  Alistair D. indicated that some
results for these kernels on the CRAY T3D had been collected.
Bodo P. asked if Strassen's method could be used? Jack D. indicated that 
the current benchmark's validation procedure may fail with Strassen's method.  
Roger H. pointed out that benchmarks must obtain answers within the same
accuracy of the standard method (used in original benchmark) and that the
flop count must not be changed either.  All attendees agreed that the
author of a benchmark gets to decide what the flop count should be.  Alistair
D.  asked about the status of the 1-D FFT.  David B. will put together
a parallel version soon using MP.

ACTION ITEM: David B. will construct parallel 1-D FFT using message-passing.

Bodo P. pointed out that codes should be able to exploit a variety
of memory configurations.  David B. said new code will do this.  Alistair D.
then asked about the status of the I/O benchmark.  Nothing was reported.
Pat W. asked if any CA's had any significant I/O activity?  David W. indicated
that check-pointing was the only significant I/O activity in the CA collection.
David B. indicated that he had a benchmark (BT) which monitors the output/
writing of arrays that could be used for the Parkbench suite.  Charles G.
also had similar benchmarks used within CRI.  Roger H. then asked for
a volunteer to construct a prototype I/O benchmark.

ACTION ITEM: David B. and Jack D. will provide two different types of I/O
             benchmarks.

Roger H. indicated that a description of these benchmarks be provided to Tony 
H. who chairs the Kernel subgroup.  David B. thought that his code might
be more of an CA rather than a kernel.  It was then decided that Jack D.'s
I/O code would be used for the kernel section of the suite and that
David B.'s would be added to the CA suite.

David W. then reported on the status of the compact applications (CA's).
He indicated that there are 4 CA's now in Netlib (NCAR/ORNL shallow water,
UK QCD,  ARCO suite, and OCEAN code).  The code characteristics of these
CA's are listed below:

QCD           -- PVM, Parmacs,
ARCO          -- PVM & NX,
OCEAN         -- seq,HPF,PVM,Parmacs,
Shallow Water -- PVM,NX,PICL.  

All of the above codes were formally submitted via application form.  Jack D.
noted the existence of several tar files which don't seem to be part of suite
- extraneous codes will be removed from the Netlib repository.  David B.
tried to get a PIC code but has had no luck thus far.  Roger H. is working on
a 3-D PIC (Particle-in-Cell) code for the USAF.   Mike B. indicated that an 
environmental modeling code (NOYELP) might be another possibility for a PIC 
code.  David W. indicated that he has a Fortran-90 3D PIC code also.

For the compiler subgroup, Jack D. (speaking for Tom Haupt) indicated
that nothing has changed.  One concern is the availability of an
HPF compiler.  Charles G. indicated that CRI is not committed to HPF
currently as it is not standard.  IBM is working on it.

At 11:21am,  Roger asked David B. about the availability of NAS benchmarks for
overseas acquisition.  New codes without comments on restricted access
will be added to Netlib (David B.).  Jack D. indicated that he has acquired a 
PVM version of NAS parallel benchmarks and will put them in Netlib. 

ACTION ITEM: David B. and Jack D. to place new versions of the NAS
             Parallel Benchmarks in Netlib.

David B. indicated that HPF versions of the NAS Parallel Benchmarks have
been done by David Serrafini (Rice Univ).  For future development,
David B. indicated that a multi-zone benchmark with overlapping grids
and a serious unstructured grid might be added.  These would be full-scale
applications rather than CA's.  New larger sizes for the benchmarks are also
being considered.   He also shared with the attendees the official NASA
statement concerning the export of the NAS Parallel Benchmarks...

``Codes used for computer benchmarking are not considered competitively 
  sensitive''

At 11:32am, Alistair D. questioned the cleanup of benchmark codes.  He felt
standardization of makefiles would be helpful and asked if a language standard
should be imposed?  Roger H. thought that the Fortran 77 standard was 
sufficient although some of the codes are not fully Fortran 77.  Bodo P.
pointed out that Fortran 77 plus extensions (DOD) are now considered standard 
Fortran 77.   It was then noted that PVM 3.2 might also be a standard to use.
David B. pointed out that no one wants to rewrite codes with more restrictions.
Alistair D. pointed out that makefiles and timer routines do vary across the
suite of codes.  Mike B. and Bodo P.  stressed that single file source is the
easiest to maintain.  Charles G. indicated that the Genesis benchmarks have too
many interactive files which should be removed.  Alistair D. asked if any
standard documentation should be used.  David W. suggested that the 
code submission form provide documentation.  The question of how timer
routines should be maintained across different platforms was raised.
Mike B. suggested that a common timer template (similar to that used by
the Perfect Benchmarks) be used.  Charles G. pointed out that TICK1 and TICK2
should be uniformly used within all the codes.  

ACTION ITEM: Jack D. and Alistair D. will define the timing routine standard
             for Parkbench codes.

Alistair D. then suggested that the number of files per benchmark be
kept small and to isolate machine-dependent code.

At noon, lunch was served, and the meeting moved to Jack D.'s on the UT campus
for demos of Southampton's GUI and Tennessee's PDS.  Alistair D. demo'ed
the Southamption GUI and Todd L. demo'ed the current PDS software. Both
tools are built using Mosaic.  The demos were completed at 2:10pm.

Roger H. then asked about the status of the BOFS's at the upcoming 
Supercomputing '94 conference in Washington, DC.  Jack D. indicated
that he had make a request for a Parkbench BOFS but has heard nothing
back.  Attendees agreed that there should be a joint BOFS with
SPEC/HPSC (which should be open to the public).  Since there would
be a total of 3 BOFS (SPEC/HPSC, Parkbench, combined), separate 
evenings will be targeted.

ACTION ITEM: Fiona S. will make a request to the SC'94 Program Committee
             for the joint Parkbench-SPEC/HPSC BOFS.

Alistair D. suggested that a hardcopy of Parkbench performance results
be distributed at SC'94 along with Mosaic instructions for the GUI/performance
graphs and PDS tools.

ACTION ITEM: Alistair D. will make specifications for submitting data to the
             database used by the Southampton GUI and PDS.
             for the joint Parkbench-SPEC/HPSC BOFS.

At 2;20pm, Roger H. asked the group where Parkbench should be headed in
the near future?  Everyone agreed that data acquisition and analysis should
consume future efforts.  Roger indicated that Southampton will conducted
research along the lines of "Parkbench Performance Profiles" in which
say 10 numbers are used to characterize parallel performance.  Fiona S.
pointed out, however, that one has to have a different performance model
for the optimized codes (i.e., data-fitting may not be appropriate). 
David W. suggested that more CA's be collected.  Charles G. remarked that
the current set is not a good random sample of application codes.  Jack D.
suggest that "good" codes be acquired and then solicit the public for
what is needed.

Jack D. then asked the attendees to list potential Parkbench organizations
and applications.  The list follows:

Ask to join:
-----------------------
LANL (Margaret Simmons)
LLNL (?)

Applications to consider:
------------------------
PIC
Structural Mechanics
Molecular Dynamics (Fiona S. has a list of public-domain MD codes)
Databases
N-body Solvers
Device Simulation (PISCES)
Astrophysics (Bodo P. may be able to check on these)
Environmental Modeling
EuroBen (Single processor benchmarks)
PEPS (Funding hiatus)

At 2:55pm, Roger H. raised the question of selecting a new chairman.
Since Tony H. indicated his willingness to be chairman (by proxy),
the attendees voted that Tony Hey of Southampton be the next
Parkbench Chairman.  Jack D. indicated that he would be willing to
be Vice-Chairman and the group accepted his offer.  Jack D. will
also investigate the video-conferencing option and send specifications
of the type of technology that might be used for Parkbench future
meetings.

Roger H. adjourned the meeting at 3:06pm EDT
---------------------------------------------------------------------

---

From: "Chuck Mosher" <[email-masked]>
Date: Thu, 1 Sep 1994 16:18:42 -0500

On Sep 1,  2:51pm, Michael W. Berry wrote:
> Subject: Minutes of last meeting

>  Alistair D.

The routines in the ARCO suite perform intense I/O operations, with both
sequential and transposed (scattered) access pattterns.  Data volumes range
from 10 MB (small) to 48 GB (huge). The ARCO suite provides a set of routines
for parallel I/O on regular multi-D arrays with support for files > 2GB and
software striping across file systems.  The routines have been tested on a
wide range of hardware and software configurations.

-Chuck Mosher

---

From: "Michael W. Berry" <[email-masked]>
Date: Thu, 01 Sep 1994 20:50:28 -0400

> wide range of hardware and software configurations.
Thanks for your comment Chuck.  I'll add this information to our minutes.
Mike

-----------------------------------------------------
Michael W. Berry                           |\---/| \\
Ayres Hall 114                             (.o o.) //
Department of Computer Science             -=~+~=-//
Knoxville, TN  37996-1301                oO(     )Oo
[email-masked]                         (_)\ | /(_)
OFF:(615) 974-3838  FAX:(615) 974-4404      OoooO
URL:http://www.netlib.org/utk/people/MichaelBerry.html
------------------------------------------------------

---

From: <[email-masked]>
Date: Fri, 02 Sep 94 11:04:30 -0400

>been done by David Serrafini (Rice Univ). 
 >

I tried finger and Rice Mosaic pages to find this person in vain. Could 
anyone help me to get in touch with David Serrafini, please?

Thanks,
Tom

---

From: [email-masked] (David H. Bailey)
Date: Fri, 2 Sep 1994 08:12:34 -0700

Try serafini, not serrafini
DHB

---

From: Pat Worley <[email-masked]>
Date: Thu, 6 Apr 1995 10:25:23 -0400

1. Benchmark Consolidation

	   Release 1.0 of the suite is to be fully tested by June 1, 1995.

           ...
	
	     Compact Applications -  (discussion began at 10:50am).
	
             ...
	
	     Enviro. Modeling - POLMP,  PSTSWM (PICL-based) 
	
	     ...  David S. also suggested to rethink PICL version of PSTSWM.
	     Mark B. indicated that programmers have had a problem with
             PSTSWM.

Could someone please remind me how CA codes are to be validated and
performance data collected? Southampton seems to have taken the lead in some
of this work, but I am uncertain as to how they are approaching it.
My concern is the following. PSTSWM was not originally developed to be a
publicly-available benchmark code, and currently has some peculiarities or
deficiencies. Some of these peculiarities are intrinsic (the large number of
runtime options), while others are due to the research motivating the code
development coming first (PICL dependence and lack of external documentation).
I am willing to modify PSTSWM to make it more suitable, and I would hope that
if people are having problems with the code, that they would 
contact me. The only problem I have heard of was from last summer (and that
was secondhand) and was due to the lack of documentation on porting the PICL
communication library. The code has been used extensively across many
platforms in our research, and problems in running it are a basic concern to
us. 

On a related point. Ian Foster, Brian Toonen, and I recently finished an
extensive benchmarking exercise using PSTSWM, measuring performance on the
Paragon, SP-2, and T3D. Can or should this data be included in the 
performance database, or does data need to be generated by an "independent"
ParkBench effort? We are also interested in "helping" run the code on the
other machines mentioned in the minutes, but do not have access to many of
the machine ourselves. (Please contact us.) 

A final few points: PSTSWM has evolved from depending on one
communication library (PICL) to running using native routines on a large
number of machines. Currently, it runs as a PICL, PVM, MPI, MPL, NX, SUNMOS,
VERTEX, SHMEM, or serial code, with the communication library chosen at
compile time. For our benchmarking, we use the native libraries and
optimize over the large number of tuning parameters to decide on what
parallel algorithm to use. The result is an estimate of how well each machine
can solve the problem rather than how well it can run a particular parallel 
algorithm using a fixed communication library. Is this appropriate for
ParkBench performance numbers? We can specify a communication protocol and
a parallel algorithm that should work across all platforms, but it will be
far from optimal, especially on machines like the T3D where the SHMEM
libraries perform much better than PVM on this code. (The message passing
semantics required by the code are very simple and the added functionality
provided by tagged message-passing is not needed.)

I am sorry to have missed the latest ParkBench meeting, but illness prevented
me from attending. I hope that these issues can be resolved by e-mail.
	
             ...
	
	     David W. indicated that Pat Worley can improve PSTSWM for
             release.

PSTSWM has been a moving target over the past year, but an improved
version is now available. (The basic code has not changed, but removing PICL
dependence and  porting to the different platforms required numerous minor
structural modifications.) I have held off updating the ParkBench version
until the new version was stabilized. What is the appropriate procedure for
updating ParkBench codes, both now and in the future? (I expect further
changes in the form of additional native ports.) There is a PSTSWM
homepage under my control where I can put new versions. How much version
control of the codes does ParkBench want or need?

        ...
	
	4. General WORK required:
	
	     3) Update and stabilize existing codes
	        April 7 deadline for contacting application experts for
                including CA's (Worley, Mosher, Guest).
        ...

I haven't been contacted (:-)), What sort of response do you want?

Pat Worley