Subject: Single Figures of Merit

From: [email-masked]
Date: Sun, 27 Jun 93 19:49:42 GMT

QUESTION TO SCHNEIDER ON SINGLE FIGURE OF MERIT
            -----------------------------------------------
I am not sure what you mean by a single figure of merit. 
(1) Is reporting many values of R_B for a set of benchmarks with
different sizes and a range of numbers of processors ( say 30 values
of R_B), reporting a single figure of merit? because one has only reported
one type of metric, i.e. R_B, and not also included other metrics like
R_H, R_T, S_p.
(2) Or does one only commit the crime when one averages the 30 values
to obtain THE overal average R_B for the computer, by taking some
average, like the SPECmark is the geometric average of many SPEC ratios?
(3) The Parkbench discussions have, so far, been based on my draft of
chapter 1. This contained no statement on averaging to get a single figure
of merit, because I do not believe that such averaging has much meaning.
And no member has proposed during discussion that we should do this. Thus by 
implication the committee does not, as yet, propose averaging
to get a final single figure for a machine. I hope that this remains the
position, but it might be useful to explicitly say that we do not support
such averaging. My view as repeatedly expressed is to show the full rang
of benchmark results graphically, 100 results= 100 dots on the graph. In\
order to plot these it is useful if they are all expressed in the same
way, and I suggest that R_B(N;p) is the most convenient metric to use
for this.