Subject: Reply to Phil Mucci regarding COMMS1

From: Roger Hockney <[email-masked]>
Date: Fri, 13 Dec 1996 13:53:49 +0000

This is a response to the contribution of Phil Mucci on the Low-level
Benchmarks:

Phil, I think you are proposing to throw far too much away and perhaps 
even "the baby out with the bath water". I think also there is a 
misunderstanding as to the purpose of the Low-level Parkbenchmarks.

I shall respond to your points as they appear in your note. Your text 
is identified with an introductory >> in the usual way:  

>>Since I started on this project, I have been complaining about the
>>poor organization of the Low Level benchmarks. 

You will have to be more specific here. COMMS1 in particular has been 
picked over by several respected people at Southampton, so there 
cannot be too much wrong with it. Remember programming style and
organisation is very subjective. One person's beautiful pogram is
another person's abomination, particularly if they come from very 
different backgrounds and experiences.

>>In fact, more than one executive has complained/asked me why they 
>>see such poor performance out of the comms. 

As detailed below the LL benchmarks aim to test the performance
of the computer as seen through the high level programming interface 
that is used. If COMMS1, e.g., gives poor results, the executive
should find out what it is in the Fortran programming interface that 
degrades the basic hardware capability which his internal tests are 
probably measuring. Users of his company's computer are more likely 
to see the performance through a high-level language and obtain 
results like COMMS1, than they are to see the performance of his
internal tests.

>>Your patch will definitely do the trick, but in the long term, 
>>a rewrite will be necessary. 

I think a minimalist approach is better. Some changes are certainly
advisable but a complete rewrite is not needed. People gradually get 
to know and understand benchmark code, and code stability is important 
to establish as far as this is possible and sensible. 

>>In addition, I believe the data 
>>analysis section should be completely separate from the benchmarks
>>themselves. The benchmarks should produce data points, and fitting
>>should be done externally to handle this kind of situation. (I'm sure 
>>something similar will arise again...)

Section 3.2 of PBR states that the " ... LL benchmarks aim to 
measure performance parameters that characterise the basic architecture
of the computer and the compiler software through which it is used".

The measurement of performance parameters requires a curve-fitting 
procedure, therefore this means that the analysis section is an integral 
part of the benchmark and should not be separated from it. Why make two 
jobs out of one? Internally of course the curve-fitting procedure is 
separated out as a subroutine. Actually this is the only subroutine that 
needs to be changed in COMMS1 to solve the difficulty that Ron Sercely 
has had with COMMS1 (see my response to his e-mail to this group). 

This is where you are proposing to "throw the baby out with the bath
water", 
because if you remove the performance parameter calculation from COMMS1 
you remove a great deal of its value. Remember that the curve fitting 
replaces a lengthy table of numbers with just two performance parameters 
(rinf,nhalf) and a formula r=rinf/(1+nhalf/n). This is a tremendous
simplification which can be used to produce timing relations for more 
complex codes in the form of mathematical formulae. Such a performance 
characterisation really does open up new oportunities for understanding
and interpreting performance. It should not be thrown away because of 
trouble with the fitting program. The best solution is to correct the 
fitting program, in the way that I have shown. 

The value of the parameter nhalf is particularly revealing because it 
tells you how important in terms of message length a particular value 
of t0 is in degrading bandwidth performance. The factor (1+nhalf/n) in 
the above formula IS the degradation factor introduced by the startup 
time. Notice the startup time itself is not relevant, it is the product 
of startup time and asymptotic bandwidth that counts, and that EQUALS 
nhalf. That is to say nhalf is the number of bytes that could have been 
transferred at the asymptotic rate during the startup time. 

For more on (rinf,nhalf) the best source is my new book "The Science of 
Computer Benchmarking" SIAM 1996 in Jack's series on Software, 
Environments, and Tools (the book's URL is on my home page, whose URL 
is in my e-mail signature below).    

>>We should start over and have them rewritten in C,

Why C? There is no point in writing a benchmark in C and thus measuring
the performance as seen through C, if you want to find out the 
performance of code written in Fortran. To find out the performance of
code in Fortran, the benchmarks must be written in Fortran.

This is not to say that a version of the LLs in C would not be welcome 
and useful to those who might be programming applications in C. It's
just that the Parkbench Committee initially selected as its primary 
programming model Fortran77 + PVM and later extended this to F77 + MPI 
(see section 1.4 of the Parkbench Report (PBR)) 

>>scrapping the output format and curve fitting code.

The present output format gives both the primary measured data (t vs n)
and the changing performance parameter values as more data is added.
This is valuable for seeing the stability of the fitted parameters.

I and Ron would like to see one column added for the measured bandwidth
(= n/t). This should have been in the original code. It was an omission
which I would like to see corrected.

I would also like to see a slight change in the presentation of the
Summary Results as indicated in my reply to Ron's comments.

Users can then decide whether they wish to just use the table of
measured 
(t vs n), or whether they wish to use the parameters. They may also
re-analyse the data in any way they wish. All options are still open.
Since the curve fitting is sometimes difficult (as we are learning) it 
is surely helpful to provide, as part of the benchmark, the best shot we 
can for this. Usually for COMMS1 the parametric fit works exceptionally 
well.

>>In addition, they should be written by someone who understands 
>>end-to-end performance issues. 

Can you explain to the uninitiated what this means, please.

>>(Remember, Low Levels are supposed to tell you what the hardware's 
>>capable of, not what application throughput you can expect.)

I think that you are quite wrong here. The Introduction of the
Parkbench report clearly states in section 1.4:

"Computer benchmarks are computer programs that form standard tests
for the performance of a computer and the software through which 
it is used ..."

and later in section 1.4:

"A benchmrk is therefore testing a software interface to a computer,
and not a particular type of computer architecture. For example 
benchmarks using the "F77+PVM" programming model can be run on any 
computer providing this interface ..."

Nowhere in the PBR does it say that the LL benchmarks are supposed 
to tell you what the hardware is capable of. From the users' 
perspective, I think the PBR's point of view is the more useful one. 

>>However, I'm sure all the vendor's would like to start seeing 
>>numbers that parallel their own internal measurements...

This is up to the vendors themselves by finding out what in their
Fortran interface is degrading the performance, and then improving 
the interface. 

This is the kind of constructive effect that we hope the benchmarks 
will have on the vendors. For example, COMMS1 has drawn the vendors 
attention to the importance of reducing t0 and nhalf in addition to 
their previous concentration only on increasing maximum bandwidth 
(=rinf).

Actually we at Southampton who originated these LL benchmarks have 
NOT heard many "complaints" from vendors about the COMMS1 results.
In our experience they usually agree with them. If there are such 
complaints, other than that from Ron, we would like to know about 
them, and can probably help you more than anyone else to resolve 
any problems. 

It is anyway right that the code originaters be contacted for 
agreement before any changes are made to these benchmarks. The code 
and revision history WAS part of the code contributed by the 
Southampton group to the Parkbenchmarks. You can tell from this 
who did what and when. It is important that this historical record 
be kept with the benchmarks. I notice that it has already 
disappeared from the output files where I think that it should be 
reinstated.   

Thank you Phil for raising these issues which are important ones. 
-- 
Roger Hockney.  Checkout my new Web page at URL   http://www.minnow.demon.co.uk
                and link to my new book: "The Science of Computer Benchmarking"
                suggestions welcome. Know any fish movies or suitable links?