Subject: speedup

From: [email-masked]
Date: Tue, 15 Jun 93 16:36 GMT

Re: Speedup.

I would like to voice my agreement with David Bailey's comment:

>>Although some of us may dislike speedup figures, I feel that like
>>Mflop/s they are here to stay.  It does not seem realistic to
>>"prohibit" researchers from using this statistic.  Also, I feel that
>>inventing a whole new statistic, "generalized speedup" (or should we
>>say "generalised speedup" for our British colleagues?), is not a wise
>>path to follow unless we have very persuasive reasons.  It would only
>>add more confusion to the field.
>>
>>Given that speedup figures, properly formulated, are a legitimate
>>performance statistic for studying the linearity characteristics of a
>>single parallel system or application, I believe that all we need to
>>do is to establish some general guidelines so that its usage is honest
>>and scientific.

Although we cannot stop anyone using any metric to justify their
comparisons, we can at least add to the PARKBENCH report a section
detailing what speedup is. ie A software metric measuring
how well the chosen algorithms take advantage of parallel 
hardware and NOT a benchmarking metric. 

As the great speedup debate appears to have come up against some issues
of R_b that have been discussed in the Methodology sub committee, I
would like to recap on those messages and a couple of further points.

Here are the main points of the original message:

>>With reference to 2.4.3 Benchmark Performance. If we are comparing the same
>>benchmark on different machines do we need Fb(N) in the equations?
>>
>>For instance R1b(N;p) > R2b(N;p) where R1b is the rate of machine 1
>>and R2b is the rate of machine 2.
>>
>>This implies Fb(N)/T1(N;p) > Fb(N)/T(N;p) as Fb(N) does not depend on 
>>the machine but only on the defining serial code.
>>
>>This implies 1/T1(N;p) > 1/T(N;p) (or T(N;p) > T1(N;p) ).
>>
>>In other words we get a straight timing result anyway.
>>
>>If we are not comparing the same benchmark is the flop a suitable 
>>representation of the work that a benchmark does? Is it not possible that 
>>memory accessing and I/O usage could outstrip floating point operations in 
>>terms of importance?

Here is Roger Hockney's reply:

>> Do we need F_B(N)
>>The answer is NO we don't, in which case the only metric to use is the
>>Temporal Performance, which is the same as comparing executions times,
>>but the result is expressed more naturally as a performance. The definition
>>of Benchmark performance was made in order to keep the units of Mflop/s
>>which people widely use but ensure that the highest R_B implies the least
>>execution time.  This can only be done if F_B(N) is kept the same for
>>all implementations and treated as a nominal figure.  If you want real
>>Mflop/s you use R_H the hardware performance.  Of course F_B is only
>>appropriate for problems dominated by floating-point arithmetic, for
>>other problems the Benchmark writer would be expected to define an
>>appropriate measure of work (I/O references e.g.) or else simply fall
>>back on Temporal performance and time alone. 
>>	The other reason for R_B
>>is to allow some (albeit approximate) comparison of all arithmetic
>>benchmarks in the same units (i.e.Mflop/s). Temporal Performances
>>cannot be compared across benchmarks.
>>           All Good points for discussion on Monday, preferable as
(If this was discussed can we see the appropriate minutes?)
>>suggested editing to my draft submission, with your agreement of course
>>David. Are you there
>>                        Roger Hockney

My reaction is immediate concern over:

>>the Benchmark writer would be expected to define an
>>appropriate measure of work

I do not believe this will simplify matters.
Further:

>>	The other reason for R_B
>>is to allow some (albeit approximate) comparison of all arithmetic
>>benchmarks in the same units (i.e.Mflop/s).

For this to be worthwhile you will need a wider definition of work, also,
given the dependence of results upon problem size, you will need
a non-arbitrary definition of problem size. We could end up with
a finite element problem size given by the number of elements compared
to a linear algebra problem size determined by N say when dealing with an
N by N square matrix.

One further point :

Your phrase ``the use of a better algorithm which obtains the solution 
with less than Fb(N) operations will show
up as higher benchmark performance.''

This implies you are allowing for algorithmic improvement. I fully 
agree with this approach but it does mean PARKBENCH must think about
how to present benchmark definitions, and how to report algorithmic
changes. In order to prevent a programming competition, we will
need to make other people's implementations available. This will
also discourage intensive optimisation by companies, if they're 
helping everybody rather than just themselves.

		Chris Francis
		National Physical Laboratory
		[email-masked]