Subject: Parkbench

From: Clark Thomborson <[email-masked]>
Date: Thu, 14 Apr 1994 10:47:48 -0500

Dear Professor Dongarra,

	Please add me to your distribution list for Parkbench.

	I'm the author of the article in the 11/93 C. ACM on porting
workstation code to vector supercomputers.  Among other projects, I am
now trying to figure out how to extend that broad-brush analysis of
the cost and performance (broadly construed) of high-performance
computer systems.  Broader benchmarks than single-user CPU or
CPU-memory kernels are surely part of the answer I seek.

	Just now, I retrieved "method.archive from pbwg" to see if
your methodology group is headed in the direction of my interest.  It
seems not...so I'll ride my hobby-horse a bit longer...

	Measurement under a vaguely-stated "load" is, of course, an
ill-defined problem, but it has the virtue of leading one to other,
even more important, questions:

1. What are the likely job mixes for any particular supercomputer?

2. Which components of the job mix are most important to accelerate?

3. Which components should be run as efficiently as possible, i.e., in
such a way as to maximize system throughput, rather than minimizing an
individual job's latency?

	It seems that the supercomputer benchmarking community has
already answered these questions in the following way:

1.  Floating-point intensive (most typically BLAS kernels).

2.  BLAS kernels.

3.  None.

	Perhaps these are indeed the "right" answers, but I'd be
interested to hear a justification, especially for Answer 3.  (Answers
1 & 2 define the traditional military/scientific market for
supercomputing; an abrupt focus-switch to an entirely different market
segment is, I believe, beyond the scope of Parkbench.)

	Answer 3 is, I believe, inappropriate.  An architecture that
is sharply tuned for performance on single-job kernels, but with a
large process-switch time, will be cost-ineffective in comparison to
an architecture that can support multiple jobs without much
degradation in single-job performance.

	To put my concern in another way: it is "merely a small matter
of programming" to hand-code any particular BLAS-3 kernel for ideal
speed under no load on any particular system for which you know the
memory and instruction latencies and throughput.  But!	I submit that
any such code is likely to have significant performance deficits when
there are memory conflicts (e.g. bank conflicts on a Cray Y) or
processor-scheduling delays (e.g. in a Cray autotasked code run on a
multi-user system).

	Surely others in the Parkbench group have noticed this
nonlinear, load-dependent, performance degradation on a Cray X-MP or
Y-MP.  And things will surely get worse, soon, when more of us write
codes using more than two layers of the memory hierarchy (as is
necessary on any current MPP, or for that matter on any vector
supercomputer chewing on a 10 GB data file).  How large of a memory
image, at each layer of the memory hierarchy, should we assume for the
purposes of performance-tuning?  Can we write our source codes and
system subroutines so that their performance depends no more than is
necessary on system load??

	Special-interest disclosure: My comments above are partially
motivated by some ideas I have been developing recently, about how to
write source codes for BLAS-3 subroutines that acheive near-optimal
performance under a wide variety of system loads.

						Clark Thomborson

PS: Thank you for organizing this email discussion group, and for
publishing the "Parallel Computing Research" newsletter.