Subject: Response to Scneider on Speedup

From: [email-masked]
Date: Sat, 5 Jun 93 21:04:15 GMT

Thank you for your contribution to the Speedup debate, particularly
concerning what went on at SPEC. Personally I think it is too defeatest
to fall back on execution time alone (my Temporal Performance), because
this means that one cannot compare, even approximatelty the performance of
one benchmark with another. Also the performance of different
sized instances of the same benchmark will vary widely, and are difficult
to plot. Whilst I think Temporal Performance should always be reported
s the primary measurement, I like the Benchmark Performance with its
flop-count that is etched in stone for all time. This has the sound
property (unlike Speedup as usually applied) that a greater benchmark
always means a shorter execution time. It also means that all benchmark
performances are measured in approximately the same units (namely Mflop/s)
which allows approximate comparisons across benchmarks, which Temporal
Performance does not. Also all problem sizes have approximately the
same performance when expressed as benchmark Mflop/s, which is a great
convenience for plotting and comparison.
  More importantly, whilst I can understand in general terms what you
and Dave Snelling are saying about an axiomatic approach, I do not
understand actually what you propose. What is the axiomatic approach
when applied to benchmarking? Can you be specific and give us some
text to consider, and proceedures to adopt? Concrete suggestions
are what we need.
Roger Hockney