Subject: Speedup and R_B Mflop/s

From: [email-masked]
Date: Fri, 21 May 93 15:41:53 GMT

May I respond to some comments on my submission on metrics:
(1) To Trevor Chambers - Speedup and Performance per Node
I think you may have talked yourself into a job! Although I personally
do not think that Speedup is a useful metric, I acknowledge its widespread
use, and agree that the best policy is to try to tighten up the definitions.
Please put your suggestions to the 24th May meeting. My problem with the
use of Speedup is that I don't think that it is possible to produce a usefull
definition, because all the knowledge of absolute performance has been
thrown away by taking the ratio to some (what) one-processor time. Its
much more rational to keep the knowledge of absolute performance by
using e.g. Temporal or Benchmark performance. How can you compare 
different computers if you don't use absolute measures, which involve
seconds^-1 in their units? Lets discuss this on Monday.
(2) To CBF at NPL - Do we need F_B(N)
The answer is NO we don't, in which case the only metric to use is the
Temporal Performance, which is the same as comparing executions times,
but the result is expressed more naturally as a performance. The definition
of Benchmark performance was made inorder to keep the units of Mflop/s
which people widely use but ensure that the highest R_B implies the least
execution time.  This can only be done if F_B(N) is kept the same for
all implementations and treated as a nominal figure.  If you want real
Mflop/s you use R_H the hardware performance.  Of course F_B is only
appropriate for problems dominated by floating-point arithmetic, for
other problems the Benchmark writyer would be expected to define an
appropriate measure of work (I/O references e.g.) or else simply fall
back on Temporal performance and time alone. The other reason for R_B
is to allow some (albeit approximate) comparison of all arithmetic
benchmarks in the same units (i.e.Mflop/s). Temporal Performances
cannot be compared across benchmarks.
           All Good points for disussion on Monday, preferable as
suggested editting to my draft submission, with your agreement of course
David. Are you there
                        Roger Hockney