Subject: Generalised Speedup

From: [email-masked]
Date: Mon, 28 Jun 93 11:48:43 GMT

REPLY to Xian-He SUN - 2
                        from Roger Hockney
                    -------------------------

Thank you for answering my detailed questions on Generalised Speedup (GS).
From these I see that TU, the unit of time used in GS, changes for 
different computers, and even for different numbers of processors on the 
same computer, i.e. the dependency is, in general, TU(A,X:N;p). Thus, I
conclude the following: 

  (1) We *cannot* conclude that if the GS on computer B is greater than
      the GS on computer A on a particular benchmark (same X:N), then 
      computer B executes the benchmark in less wallclock time than 
      computer A. Because the relationship is:
      if               GS(B,X:N;p)    >    GS(A,X:N;p)
      then by definition
                       TU(B,X:N;p)         TU(A,X:N;p)      
                      -------------   >   -------------
                        T(B,X:N;p)          T(A,X:N;p)
      or
                                 TU(B,X:N;p)
                  T(B,X:N;p) <  ------------- * T(A,X:N;p)
                                 TU(A,X:N;p)
      and, unfortunately, the ratio of TUs is not guaranteed to be unity,
      because the value of TU depends on the computer. This is also true
      for conventional Self-Speedup.
      For an example take: TUB=9, TB=4, TUA=3, TA=2, 
          then  GSB>GSA and although TB<3*TA, WE HAVE IN FACT TB>TA
      However, the ratio of TUs is, of course, unity for the metrics R_T, 
      R_B because in these cases TU does not depend on the computer or 
      the number of processors. Therefore the R_T and R_B metrics do not 
      suffer from the above problem.
     
  (2) Because of (1), the ordering of computers in order of GS value is 
      not necessarilly the same as the ordering by inverse wallclock time
      which is what, I believe users and the committee expect. R_T and
      R_B, of course, do retain the inverse wallclock time ordering. 

  (3) Looked at in another way, the GS metric uses different time units
      for different computers. Such numbers cannot be directly compared
      across computers, and to do so is like comparing the numerical values
      of speeds of cars measured in different units, m.p.h., f.p.s, cm/s,
      in order to decide which is the fastest. In both cases such comparisons
      are not comparing like with like, and are invalid. More confusing
      still, the time unit changes with the number of processors, making
      GS values incomparable even for measurements on the same computer 
      for different numbers of processors.

  (4) There are undoubtedly other good reasons for computing and studying
      values of GS, for example in the understanding of so-called 
      "super-linear" speedup. However in view of points (1) and (3) above,
      I do not believe that GS is a suitable metric for the Parkbench 
      committee to use to report benchmark results. Although there is no 
      reason why others should not compute such metrics for their own 
      purposes from the data provided by Parkbench.

                             Roger Hockney