Subject: Reply to SUN - 3

From: [email-masked]
Date: Sat, 3 Jul 93 20:43:53 GMT

Reply to Xian-He Sun - 3
                   ------------------------
                      from Roger Hockney
                      ------------------

(1) I take your point that for the same problem size TU does not depend 
on the number of processors. It does however, as you agree, depend on the 
computer, and this is the source of the problem. If we wish to report the
actual (as opposed to potential) performance of computers, then GS is not
a suitable metric for ParkBench results , because a higher GS does not 
necessarilly imply a shorter wallclock time. Again your reply to my point
(1) shows that you agree with this, indeed GS is not intended to measure
wallclock time, it is intended to measure Speedup which is something 
quite different.

(2) You misunderstand me about the units: Of course TU(A) and TU(B) are 
both measured in the same units, say seconds. However the numerical value 
of the ratio T(A)/TU(A) is the time on computer A measured in units of
TU(A): it is how many TU(A)s there are in T(A). Similarly T(B)/TU(B)
is the time on computer B in units of TU(B). The inverses of these two 
quantities are GS(A) and GS(B). So that comparing GS(A) with GS(B) is
like comparing two inverse time measurements, in which the units used
to measure time are different, because TU(A).NE.TU(B) in general. So that 
if one is interested in comparing wallclock times, one is comparing the
numerical value of two measurements that are expressed in different units 
(like the speeds of my cars in m.p.h. and cm/s). Such a comparison is
invalid because one is not comparing like with like. However, of course, 
if one is not interested to compare wallclock time, but interested in 
comparing Speedup (generalised or not) for its own sake, then one may 
compare GS(A) and GS(B). 

(3) My point is that ParkBench should keep its feet firmly on the ground 
and only compare actual observed performance of computers using metrics 
that, for a given benchmark and problem size, preserve the performance 
ordering given by inverse wallclock time. I do not believe that we should 
confuse the issue by reporting such metrics as Speedup and Generalised 
Speedup, that do not satisfy this condition, and express some potential 
rather than actual performance. What is the use of a high speedup, if 
the processors are so slow that the actual performance is poor? True in
theory one could say we can easily speedup the individual nodes to remedy 
the situation. But this only works if the communications and latencies are
similarly speeded up and reduced. This is rarely possible or done.
The only safe policy for ParkBench is to stick to actual performance,
and use metrics satisfying the condition above. Thus, in my opinion,
we should not use Speedup or Generalised Speedup. I am not saying that
GS is not an interesting and useful concept, far from it; I am saying 
only that it does not measure directly actual performance, and is 
therefore unsuitable for expressing benchmark results which are supposed 
and assumed by readers to report actual performance. 

                        Roger Hockney