Subject: Notes on HINT and QUIPS

From: "Dave Snelling" <[email-masked]>
Date: Wed, 2 Nov 94 10:21:27 GMT

John, Quinn, and PARKBENCH,

I am making these comments (constructive I hope) openly in order to
disseminate the ideas as quickly as possible. I believe that, whether right
or wrong, QUIPS and HINT are likely to catch on. The problems below should
therefore be rectified quickly.

Problem 1:

The document describing HINT and the source code claim that the list of
intervals is sorted. Even with the well behaved function integrated in the
HINT benchmark, the relative difference between the errors of consecutive
intervals ranges from plus to *minus* ten percent by the 500 interval mark.
I assume that this trend will continue.

Possible Solution:

Correct the documentation to indicate that the list is only partially sorted.

Problem 2:

The Net QUIPS computation is heavily weighted toward the speed at which the
smallest case can be computed. For example, by treating this case specially
and returning the answer almost immediately (a very simple optimization not
prohibited by the rules), the Net MQUIPS of a Sun SPARC station improved
from 0.741 to 1.615.

Possible Solution:

Start the Net QUIPS at something more reasonable, say 32 intervals rather
than two as it stands now.

Problem 3:

By tuning the parallel version on the KSR to select the number of
processors actually used (including the special case from problem 2 above),
it was possible to obtain similar variations in the apparent performance of
the KSR. This tuning operation basically eliminates the low QUIPS ratings
associated with start-up on small numbers of intervals in parallel
configurations. Further improvements are also possible by varying the
ADVANCE parameter.

Possible Solution:

Most of these problems can be eliminated by stating exactly how many
intervals are to be used at each stage of the Net QUIPS computation.
  
Problem 4:

There is no direct verification of the answers. The value of the answer is
of course included in the value of the QUIPS rating, but for small numbers
of intervals, where the error is large anyway, random numbers returned very
quickly could artificially improve results.

Possible Solution:

Check the actual results. Unlike many benchmark programs HINT's results can
be checked to the last bit; why not do it?

Problem 5:

As it stands, memory use in parallel tests increases simultaneously on all
processors. The result on the KSR is that, when testing configurations
smaller than the whole system, one level of the memory hierarchy is
ignored.  On VSM systems like the KSR, the memory of other processors can
be used as "paging" space instead of disk. A more gradual increase in
memory use, for the parallel cases, would allow the benchmark to detect
this level of the hierarchy.

Possible Solution:

In the parallel version, increase memory use in the same way as in the
sequential case.

Conclusions:

I hope that these comments promote further discussion. I believe the quest
for a unifying benchmark is futile, but I see no reason for not trying.
Good luck.

Take care:

   Dr. David F. Snelling            [email-masked]
   Centre for Novel Computing       44-61-275-6134 office
   Department of Computer Science   44-533-708636  home
   University of Manchester         44-61-275-6204 fax
   Manchester, UK M13 9PL