Subject: DHB's two-bits' worth on speedup figures

From: [email-masked] (David H. Bailey)
Date: Thu, 10 Jun 93 10:56:50 -0700

Could I add my 25 cents' worth to the great speedup debate?

Although some of us may dislike speedup figures, I feel that like
Mflop/s they are here to stay.  It does not seem realistic to
"prohibit" researchers from using this statistic.  Also, I feel that
inventing a whole new statistic, "generalized speedup" (or should we
say "generalised speedup" for our British colleagues?), is not a wise
path to follow unless we have very persuasive reasons.  It would only
add more confusion to the field.

In my view, the concern frequently expressed in this forum about
speedup figures being used to compare systems is misplaced.  In my
voluminous file of scientific papers with abusive performance
practices (relax: I don't believe any of you have a paper in my file),
and in my reading of conference proceedings and journal articles
during the last few years, I have NEVER seen an instance where someone
used speedup figures to claim that one system is faster than another.
The only instance remotely close to this is when a scientist, based on
two separate speedup analyses, asserts that one system has more nearly
linear speedup characteristics than another.  What is wrong with this,
provided it is backed up with proper data, and the author makes it
clear that he or she is not comparing absolute performance?

Some in this forum have suggested that the single processor timing
must further be based on the best practical serial algorithm, even if
other algorithms are used for the parallel timing.  While this
suggestion has some merit, I personally feel that it is of much higher
priority for us to emphasize such strict procedures for computing
Mflop/s rates, which are frequently used to compare different systems,
than for speedup figures, which are not used to compare different
systems (in absolute terms).  However, I could be talked into making
this recommendation for speedup figures also, if others agree.

Given that speedup figures, properly formulated, are a legitimate
performance statistic for studying the linearity characteristics of a
single parallel system or application, I believe that all we need to
do is to establish some general guidelines so that its usage is honest
and scientific.  In the proposed guidelines that I presented in a
recent paper and in my talk at Supercomputing '92, I included the
following item regarding speedup figures:

If speedup figures are presented, the single processor rate should be
based on a reasonably well tuned program without multiprocessing
constructs.  If the problem size is scaled up with the number of
processors, then the results should be clearly cited as ``scaled
speedup'' figures, and details should be given explaining how the
problem was scaled up in size.

In the light of the discussion in this forum, it may be useful to add
a phrase to this statement to insure that speedup figures are not
abused as an absolute performance comparison statistic.  Other minor
adjustments might be in order.  But it general I feel it would be
fruitless to include lengthy and highly technical directives or
otherwise we may introduce more confusion than we alleviate.

Comments?

David H. Bailey
[email-masked]

---

From: [email-masked] (David John Schneider)
Date: Thu, 10 Jun 93 14:11:45 CDT

>   Date: Thu, 10 Jun 93 10:56:50 -0700
>   From: [email-masked] (David H. Bailey)
>

Unfortunately, I believe that a new low has been reached.  It is my
understanding that several million dollars has recently been spent for
a new machine at an NSF National Supercomputer Center, justified
largely on a comparison of speedups.  It is scary to think that
decision makers, not just the masses, are so easily deceived.  

>	[stuff removed]

		Dave Schneider 

		University of Illinois at Urbana-Champaign 
		Center for Supercomputing Research and Development
		367 Computer and Systems Research Laboratory
		1308 W. Main Street
		Urbana, IL  61801-2307
		MC-264

		phone : (217) 244-0055  
		fax   : (217) 244-1351
		E-mail: [email-masked]