Subject: Reply to R. Hockney

From: David John Schneider <[email-masked]>
Date: Sat, 26 Jun 1993 10:18:22 -0500

Roger,

I agree that the example that I constructed is artificial in the sense
that the performance of different codes on different machines are
simply not comparable.  The fallacy lies in trying to construct a
single figure of merit, not in the definition of F_B.  

In the case of computer performance, the definition and dissemination
of a single figure of merit encourages the comparison of incomparable
data.  In my example, the single figure of merit was the arithmetic
mean of the performance of the two codes.  Other means and
normalization conventions, such as the one used by SPEC, have the same
basic problem.  The basic problem indeed very basic -- a single figure
of merit defines a total ordering on the set of machines while, in
reality, elapsed time measurements on a set of codes define only a
partial or topologicaI ordering on the set of machines.  The point of
my example was the uncertainty inherent in an arbitrary but
"reasonable" convention for defining F_B can lead to different
orderings specified by a single figure of merit.

My concern is primarily related to the statements made by several
people about the desirability of the PBWG reporting a single figure of
merit, and only secondarily with a precise technical definition of
F_B.  I have been following the discussions here rather closely, and I
haven't seen an explicit statement that the PBWG does *NOT* intend to
publish a single figure of merit.  If such a decision is made by the
group, then the precise definition of F_B becomes much less
important.  I would encourage the PBWG to adopt the policy of refusing
indirectly endorse *ANY* single figure of merit by publishing such
numbers.  

				Dave