Subject: potential compact benchmark

From: [email-masked] (Pat Worley)
Date: Thu, 16 Sep 93 11:19:06 EDT

Ian Foster and I are just finishing version 1.0 of PSTSWM, a parallel
algorithm testbed and benchmark code developed for the climate modelling
community. It will be made available to this community via netlib, but it
may also be interesting as a PARKBENCH compact application. There are a
few difficulties with this though, and I would like some
feedback/suggestions on how to proceed.

Description
-----------
PSTSWM is a parallel implementation of a serial code (STSWM 2.0) written
by Jim Hack and Rudy Jakobs at NCAR to solve the shallow water equations
on a sphere using the spectral transform method. It was originally
developed as a numerical algorithm testbed, to allow comparison of
spectral methods with finite difference methods with finite element
methods, etc., and has 6 runtime-selectable test cases in the code.
These test cases specify initial conditions, forcing, and analytic
solutions (for error analysis), and were chosen to test the ability  of
the numerical methods to simulate important flows phenomena.

For PSTSWM, we completely rewrote STSWM to add vertical levels, in order
to get the correct communication and computation granularity for 3-D
climate codes, and to allow the problem size to be selected at runtime
without depending on such nonportable features as dynamic memory. 

PSTSTWM is meant to be a compromise between paper benchmarks and the
usual fixed benchmarks by allowing a significant amount of
runtime-selectable algorithm tuning. Thus, the goal is to see how
quickly the numerical simulation can be run on different machines
without fixing the parallel implementation, but forcing all
implementations to execute the same numerical code (to guarantee
fairness). To enable this PSTSWM supports:

a) 4 classes of parallel algorithms (distributed or transpose
   based for each of two major parallel phases)
b) each class has 3-4 specific parallel algorithms (e.g. using a
   recursive-halving vector sum, using a pipelined ring vector sum,
   etc.)
c) each algorithm has 2-4 variants 
d) each algorithm is built on top of two communication constructs,
   swap and sendrecv, and each of these has 5-6 different communication
   protocol options (synchonous, blocking, nonblocking, forcetypes,
   etc.)

We are quite happy with the code, and are getting good results with it.
Most interesting to us is how the best algorithm changes across
platforms and as the problem size changes on the same platform.

Problems
--------
There are couple of issues to be dealt with in using this code as part
of PARKBENCH.

1) The code currently is in single precision with double precision
   parts. Single precision is sufficient for the problem sizes of
   interest, but the Legendre polynomial values and Gauss quadrature
   weights and nodes must be calculated in higher precision. For larger
   problem sizes, double precision computation will be appropriate, but
   the Gauss weights, etc will then need to be calculated in quad.
   precision. I do not think that this sort of mixed case has been
   discussed yet. 

2) In one sense, PSTSWM is not a single benchmark, but many of them.
   We can fix the problem and parallel algorithm specifications by
   providing (a set of) default input files, but which ones should we
   chose? All of them are arguably good algorithms in some setting, and
   I would hate to compare two machines when the algorithm is good for
   one and inappropriate for another.

3) PSTSWM is currently written using PICL (because that is what I
   normally use and because I have embedded instrumentation in the
   research version of the code). I made a real effort to isolate the
   message passing bits, so porting to anything else will be trivial.
   But the message passing interface that is used does effect the
   parallel algorithms that are supported. For example, PICL supports
   nonblocking send and receive and passes through forcetype message
   types. These are important to performance on some Intel machines.
   This is not a problem so much as something to be aware of. PSTSWM
   will also be available in its original form, but a pointer to some of
   the issues in cross-machine comparisions should be made. This may be
   an issue that should be mentioned in the methodology section as
   pertains to compact applications. Unlike low level benchmarks,
   compact applications are less likely to be "done right" by the vendor
   for their particular machines. 

Comments and suggestions would be appreciated. I imagine every proposed
compact application will be unsuitable in one form or another when it is
first submitted, and precise guidelines on what should or should not be
permitted is important. On the other hand, as a developer, I will not be
interested in doing too much work in modifying the code in order to
include it in the benchmark suite. Even with the best intentions, it
will not be a high priority item for me and is likely to be put off
(forever) if not fairly simple.

Thanks.

Pat Worley